What Happened With Tyler Technologies
First off, thanks for taking the time to read this.
If you're here, you're probably already familiar with what's happened.
If not, check out here.
If you're already familiar with all that or aren't interested in reading through it, here's the setup:
— Sat, Feb 26th around 7pm, California State Bar puts out the press release here
announcing that it is taking urgent action against a breach and unlawful display of 260,000 nonpublic State Bar attorney discipline records. Forensics experts have been retained, law enforcement has been contacted, the software vendor has been contacted, my domain registrar and web host have been notified.
— Sat, Feb 26th around 7:30pm, news websites begin picking up the story, the first of which I saw here
and the corresponding tweets here
, hashtag #hackers on each.
And I run judyrecords.
I see these around 11pm, along with several other news websites already picking up the story, immediately stop browsing reddit, and start rifling through a temp file for the closest thing that could delete
the CA State Bar cases off the index.
Meeting With California State Bar. Monday Morning.
The night I learned of the issue, I'd immediately called out the CA State Bar saying that all the records accessed were publicly available (confidential & non-confidential).
By Monday morning, however, the story had already hit local and national media.
In what can only be described as simultaneously the most bizarre and unbizzare meeting I have ever had, by the time the Monday morning meeting arrived,
everyone already knew what the problem was. Direct case access did not perform any access control check before returning case data.
Holy. Fucking. Shit.
It's one of those things that almost doesn't compute, honestly.
A security measure so fundamental, and without which the system can't even be called secure.
There's no buildup or grand reveal on the technical side of things, if you were hoping for one.
You might be wondering then, what's the most reliable way to size this up? What information available would have the highest liklihood of accurately reflecting what is actually true?
And I would simply suggest looking at what happened in the following days.
Tyler starts taking their portals offline accross the county.
I disable the search on judyrecords, and disable all Tyler Technologies cases from direct access.
The Great Retraction
Given the severity of the software defect, I'd like to think it was an entirely foregone conclusion about what the
Bar's response would be after the initial meeting.
The California State Bar released the following statement that same day:
The State Bar has continued its investigation with the help of an IT security firm, and has also been in contact with the owner of the judyrecords site, who has been very responsive and collaborative. It is now the State Bar’s belief that there was no malicious “hack” of its system. Instead, it appears that a previously unknown security vulnerability in the Tyler Technologies Odyssey case management portal allowed the nonpublic records to be unintentionally swept up by judyrecords when they attempted to access the public records, using a unique access method. The State Bar is working with Tyler Technologies, the maker of the Odyssey system, to remediate the security vulnerability, which we believe may not be unique to the State Bar’s implementation and could impact other users of Odyssey systems.
Within the span of 48 hours, I had metamorphisized from an ordinary citizen to a hacker. And to an ordinary citizen, once more.
The least biased reporting I've found up to this point here
Requests, Requests, Requests
At this point, you're undoubtedly wondering about this "unique access method" described above.
Any funny business going on there?
The short answer, is yes.
But only from the court systems themselves, actually, not from me.
Broadly speaking, there are 2 kinds of public data systems:
(1) Public data systems that make their public data available and accessible.
(2) Public data systems that make their public data available and accessible, while simultaneously and intentionally errecting barriers that make simple and straightforward access difficult.
Two-faced systems, if you will.
adjective: marked by deliberate deceptiveness especially by pretending one set of feelings and acting under the influence of another
synonyms: Janus-faced, ambidextrous, deceitful, double-dealing, double-faced, double-tongued, duplicitous, dishonest, dishonorable
Type 1 systems
— Oklahoma Statewide Court System System
— United States Patent and Trademark Office System
— United States Trademark Trial and Appeal Inquiry System
— Clayton County, Georgia Court Records Inquiry
— Arkansas Statewide Courts System
— The Supreme Court of the United States
What makes these and others Type 1 systems?
Look at the URLs
Seriously, take a look at these URLs.
Go ahead, open some new tabs, and check them out. By the way, I mean the URLs themselves, not the pages.
And what do they all have in common?
They all present simple and straightforward mechanisms to access data.
Public records portals allowing simple and straigtforward access to public records. Great!
And Type 2 systems?
Well, in Type 2 systems you have garbage like this:
What. The. Fuck.
This is from organizations that are in the same class of entity
as the Type 1 systems and URLs given above.
Each of the URLs directly above, with less
effort, could have been:
But the programmers of these systems went out of their way to intentionally create barriers that make simple and straightforward access to public records difficult.
In each of the cases above, a level of indirection was added to make the collection of public records as difficult as possible.
This is a common anti-scraping technique, which is justifiable for those with proprietary data that aren't funded by taxpayer dollars, but has no place among publicly-funded public records systems.
But wait, more Type 2 public records systems glory!
judyrecords, YouTube, and others use this same technique, in fact. See
In addition to the above scheme, you also see garbage like this:
(1) Date filed searches are arbitrarily restricted, preventing the ability to simply query all cases filed within (even a small) date range and get all the cases.
(2) Want to get an update to a case or download a public case document? Fill out a captcha first to request the case URL.
(3) Want to get all the cases of a type, filed within a certain date range? First specify at least 2 characters of the last name, and one first name character.
I've just checked 10 random Odyssey Portals and not a single one allows doing a simple date filed search.
Each one requires that additional information be supplied.
Are you seeing a pattern here?
In terms of public records access, this is intentionally antagonistic.
Records Wide Open
Tyler, as it happens, sells Type 2 public records systems.
They've implemented every single tactic mentioned above to varying degrees across their portals.
Honestly, you just deal with it.
These systems and the people who sign off on them certainly have a special place in public records hell, but you take
what you can get, and move on.
I'd pulled data from about 20 of Tyler's Odyssey Portals from jurisdictions around the country. California, Georgia,
Texas, and so on, marshalling through searches, farming out the captcha solving to a third-party service when needed.
Another day in public records aggregation paradise.
In Tyler's Odyssey Portal, cases could be requested directly, no need to jump through hoops for that part.
More often than not, cases can be requested directly, but sometimes you have to do a search first, sometimes you need to fill out a captcha, etc.
It depends on how far Type 2 they've made the system, but in this case, direct requests were available.
So, remember that first set of garbage URLs from Type 2 systems above?
The second one was actually one of Tyler's, corresponding to whatever case ID.
I remember looking at the table of a portal I'd already pulled from, and thinking:
Huh, maybe they're using the same URLs for the same case IDs?
And it turns out, they
At this point, it's just a matter of marching through case IDs, as usual.
If you've followed all the way through to this point, you might've figured out what went wrong.
Tyler went out of their way to implement Type 2 measures, intentionally requiring convoluted access approaches
by their own design, but didn't bother with basic security.
Did I ever think a direct case request could lead to private records?
Not in a million years.
Because that would mean the software had a defect so severe that any and all systems running that software
would need to be taken offline immediately, due to its inability to perform arguably its most critical function.
Well, that happened
And I downloaded from about 30 different Tyler Odyssey portals this way.
The California State Bar times 30 — potentially.
It Gets Worse
There's an old Internet adage:
On the Internet, nobody knows you're a dog.
Tyler's software didn't know either.
If you had a link, you were a fully authorized user — tail or otherwise.
I pointed out early on that this isn't just a judyrecords problem.
If these links are not performing access control checks, if a case is found by a
docket aggregator, and then goes private and has updates, those updates that happened
after the case became private are still public and subject to be downloaded.
That would seem to be a large and widespread problem, but simply hasn't been noticed yet.
Docket aggregators have millions of cases from this software system. Probably no less than 10 million on average.
According to Tyler's own
this software system serves more than 600 counties across 22 states — although I believe this counts their legacy Odyssey system as well, which doesn't have the same issue.
Two days in, I had two asks for Tyler.
My exact message:
(#1 by Thursday)
#1. Publicly clarify: Based on Tyler's understanding of the situation, what is Tyler's public position regarding whether it views the incident the same way as the CA Bar or not.
#2. Willing to budget dedicated internal staff/resources toward helping individual portals get their data up to date on judyrecords *and* other docket aggregators.
Solely from docket aggregators using URLs that, when those URLs become confidential, those URLs are still public to have data updated from (and those URLs having been this way for probably the entirety of the Odyssey case management system's lifetime), that is more than enough for Tyler to reach the same conclusion and start shepherding a unified data fix solution that can be checked and updated by other docket aggregators.
(other known docket aggregators: DocketAlarm, LexisNexis CourtLink, West's docket product, Bloomberg, UniCourt)
Type 2 Collaboration And Disbelief
By Thursday of the same week I had initially met with the California State Bar, I had already
shared exactly how I downloaded cases from their portals.
Some by search first, then direct access. The others just by direct access.
By Friday, it was great to clear the air. We discussed leveraging the same ability to identify which cases
were actually viewed that the CA State Bar used. That there are far fewer cases intended to be private that
have been viewed vs. the number that have been downloaded is an incredibly tempering circumstance.
judyrecords had also saved all case identifiers for every case downloaded, which would allow accurate
cross-referencing between systems. I've done this for every system where possible, not just Odyssey.
A strong point of contention was exactly how data would flow. I wanted data to flow from Tyler to me first,
and Tyler wanted the opposite.
When data flows from my end first, I have no ability to know what's what, Tyler
can say whatever it wants about the data, and I don't have any basis to know, evaluate, or say anything.
They control the information entirely. My main concern however, was that data flowing from me first would kill
their incentive to collaborate back in kind.
But by the end of Friday, still in the very first week of it all, I was already sending them data up front.
I did manage to require that I wouldn't be sending more than 5 sources before they had to send data back before
getting more. But I gave in to the next special request, and the next, and the next. By the middle of the next week,
I had provided 15 data sources up front.
Then, I went out of my way to create truncated files with actual data for every direct access portal.
10k-record files that could be used for testing, and for any data source for which I haven't given them a full file
But two weeks later, after significant back and forth, and despite the nature of the software defect, they are unwilling
to even say they believe the access to records meant to be nonpublic was inadvertent.
And I've only received one data file back of nonpublic case IDs to remove.
Where Do We Go From Here?
As far as removing cases on judyrecords retrieved from Odyssey (non-legacy) portals that were meant to be nonpublic,
I only need a list of case IDs. After that, a confirmation check on a sample of the cases provided, and confirmation
of the records removed.
As for general cleanup, the only viable solution I see is for Tyler to establish an API on their public records portals
so that docket aggregators can voluntarily synchronize their datasets to remove currently nonpublic cases. This will
have the effect of handling cases that became private, but updates were received after that.
After that, I seriously hope they'll consider getting rid of Type 2 designs that intentionally errect barriers
that make simple and straightforward access to public records difficult.
Supplemental Q&A (last updated 4/29)
Which versions of Odyssey Portal were affected by the direct access issue?
Tyler has 2 major Odyssey products that I know of. Odyssey "New" (referred to as just Odyssey Portal now) and Odyssey "Old", their legacy
First one looks like this, second like this.
I've asked Tyler multiple times.
I finally caved.
And just asked: What's the complicated part?
My exact words:
...confirming whether any portal has a direct access issue should be a straightforward check that could be done by level 1 or level 2 support for any of the portals.
If you don't mind me asking, what is the complicated part about verifying the authorization issue?
1) Go to a case in a portal that is supposed to be non-public
2) Open the web tools network tab and copy the URL for the case
3) Open another browser and go to the URL. If the URL displays the case, there was no authorization check.
So right now, my understanding is that any non classic Odyssey Portal had the issue of no authorization check for direct access.
It's like talking to a brick wall.
Is sharing the access method or case IDs security sensitive?
Tyler sent this
Response to #1.
No, absolutely not.
First, access method.
Tyler has already taken their public portals offline across the country.
If there are any portals that have either
(1) been taken offline then brought back online without fixing the issue
or (2) have the issue but have not been taken offline, that would be extrordinarily incompetent.
If this is the case, there are even bigger problems with Tyler.
Second, case IDs or URLs aren't sensitive information in any practical sense. Only the lack of access control
upon those, is what can make them insecure.
I just checked Washington's Odyssey system, just for the heck of it. Both identifiers exist in the case right now.
Does that mean the system is insecure? No! It's entirely irrelevant.
As mentioned previously, judyrecords, YouTube, and others use the same
type of URLs as Tyler's Odyssey system.
In fact, here's an export of internal case ID mapping to URLs for judyrecords for a currently restricted datasource — 569k cases total!
Here's the format:
These cases were blocked at the beginning of the month. (March)
Am I any less secure now?
I've implemented access control.
Those cases are for my eyes only.
It does not matter whether you have the case IDs or URLs.
Does not fucking matter.
Tyler not doing this is the exact cause of the current situation.
Exact fucking cause.
That Tyler thinks these IDs are security sensitive shows an unbelievable lack of understanding of security,
and of access control specifically.
Is Tyler Technologies a Type 2 collaborator?
Tyler sent this
Response to #2.
"expect to get you some additional files back within the next couple of days" — 3/9
"we should be able to turn that around in relatively short order" — 3/9
"for clarity's sake, we're ready to pull the trigger" — 3/9
"6 clients that we are actively testing with" — 3/6
Those are the only timetables I've known about.
Kinda makes you feel like you're being hung out to dry.
And a company has a defect in their software so bad, that by the admission of their own
is enough to warrant immediately
taking everything offline, but you can't be given the benefit of the doubt of relying on that circumstance
not being true? And if it weren't true, the code would have retrieved exactly the information intended?
I don't know how you can use the word responsible, not owning up to the issue, then leaving doubt to be cast on me for
this entire clusterfuck.
Tyler has said that judyrecords needs to provide full data about cases up front, and that this is necessary. Is that true?
Tyler sent this
Response to #3.
That's an absolute lie.
Data in this situation can flow 2 ways and achieve the exact same end result:
— judyrecords to Tyler
— Tyler to judyrecords
There is absolutely nothing in the way of achieving the exact same result when Tyler sends data to judyrecords first.
How does it look like when data comes from judyrecords first?
1) judyrecords provides all case IDs to Tyler for a datasource, with supplemental view data
2) Tyler returns back intersection of case IDs of cases to remove
How does it look like when data comes from Tyler first?
1) Tyler provides case IDs to remove
2) judyrecords returns back intersection case IDs to Tyler, with supplemental view data
As of 3/26, I've told Tyler I will no longer be providing any data sources upfront, due to their extrordinary
dishonesty and incompetence in the whole situation. Only in return.
Also see the questions:
- Is sharing the access method or case IDs security sensitive?
- Which data sources were provided to Tyler upfront?
How bad was the defect, 1 to 10?
Case access is the most critical point to ensure access control.
Everything else is a distant second. Security would easily be the most important non-functional
software handling sensitive information. Easily 10.
Does Tyler Technologies have a page for this issue?
Yes, this page
whispered its way onto the Internet about 2 weeks ago.
Interestingly, it was just updated today. (3/21)
Earlier version here
How many Odyssey Portal cases meant to be private does judyrecords have?
At this point, I believe this number is well into the millions.
These cases have been accessible over roughly the past year.
Also, this is not considering the "any dog with a link" update issue described in the section "It Gets Worse".
Where do things stand with the California State Bar?
While the investigation is considered ongoing, the California State Bar has stated:
The State Bar plans to notify complainants, witnesses, and respondents whose names appeared in the approximately 322,525 confidential records that were available on judyrecords during the period in question, though the current evidence suggests that only 1,034 of those were actually viewed. We continue to investigate this incident and will provide additional details about the notification plan and timeline as soon as possible.
The majority of Cooley’s rates and costs will be covered by the State Bar’s insurance. The State Bar and its carrier intend to pursue reimbursement from Tyler Technologies.
I'm a CIO of a potentially affected jurisdiction. How can I contact judyrecords?
You can email me at firstname.lastname@example.org.
I've provided information for numerous portal sites to Tyler upfront.
I'm sure you couldn't care less about that, and I wouldn't either, honestly.
That said, I have never had any reservations about providing data, except that it will disincentivize returning data in kind.
Toward that end, the best way to move forward is just to provide a list of case IDs of non-public cases that can be checked against.
After that, I'd be happy to provide all relevant data back about those cases, and remove them.
Tyler has taken the position that judyrecords not providing data first has held things up somehow. I cannot tell you how
incredibly frustrating and outright dishonest that is.
I'd be happy to help in this situation. Basically, you need to write a query that will return all non-public cases. Writing this query may not be just
a plain query with a single filter, so it may be necessary to reach out to Tyler technical support to those
who have worked on similar queries already. Obviously, you don't need judyrecords data to make this happen and
move things forward. Two Four Five
Six jurisdictions have done so already.
Which Tyler Technologies Odyssey Portals were accessed by search, then direct case requests?
— Alameda County, California - Court Records
— Brazos County, Texas - Court Records
— Chambers County, Texas - Court Records
— DeKalb County, Georgia - Court Records
— Fresno County, California - Court Records
— Glenn County, California - Court Records
— Glynn County, Georgia - Court Records
— Gwinnett County, Georgia - Court Records
— Harris County, Texas - Court Records
— Houston County, Georgia - Court Records
— Hunt County, Texas - Court Records
— Kansas - Statewide Court Records
— Kern County, California - Court Records
— Napa County, California - Court Records
— Ohio - Court Records
— Rockwall County, Texas - Court Records
— San Bernardino County, California - Court Records
— Santa Barbara County, California - Court Records
— Santa Clara County, California - Court Records
— Wichita County, Texas - Court Records
— Yolo County, California - Court Records
Which Tyler Technologies Odyssey Portals were accessed by direct case requests?
Case IDs of cases meant to be nonpublic have been provided
Case IDs of cases meant to be nonpublic have not been provided
Current progress as of 5/20:
Odyssey Portal cases removed:
California State Bar - Court Records
— 322,525 cases removed (Mar 2nd)
Forsyth County, Georgia - Superior & State Court Records
— 259,473 cases removed (Mar 9th)
Bell County, Texas - Court Records
— 286,200 cases removed (Mar 30th)
San Mateo County, California - Court Records
— 18,691 cases removed (Apr 5th)
San Diego County, California - Court Records
— 124,943 cases removed (Apr 13th)
Santa Cruz County, California - Court Records
— 30,381 cases removed (Apr 13th)
Tehama County, California - Court Records
— 27,526 cases removed (Apr 13th)
Bexar County, Texas - Court Records
— 105,136 cases removed (Apr 13th)
Sutter County, California - Court Records
— 946 cases removed (Apr 17th)
Yuba County, California - Court Records
— 1,136 cases removed (Apr 17th)
Mendocino County, California - Court Records
— 20,859 cases removed (Apr 17th)
Butte County, California - Court Records
— 1,193 cases removed (Apr 21st)
Stanislaus County, California - Court Records
— 6,505 cases removed (Apr 21st)
Kings County, California - Court Records
— 9,146 cases removed (Apr 21st)
Calaveras County, California - Court Records
— 5,513 cases removed (Apr 27th)
Washington - Statewide Court Records
— 12,151 cases removed (Apr 27th)
Sonoma County, California - Court Records
— 409 cases removed (Apr 29th)
Merced County, California - Court Records
— 151,790 cases removed (Apr 29th)
Chatham County, Georgia - Court Records
Dallas County, Texas - Court Records
Ector County, Texas - Court Records
Kane County, Illinois - Court Records
Lowndes County, Georgia - Court Records
Lubbock County, Texas - Court Records
Maine - Statewide Court Records
Muscogee County, Georgia - Court Records
New Hampshire - Statewide Court Records
Potter County, Texas - Court Records
Rhode Island - Statewide Court Records
Shelby County, Tennessee - Criminal Court Records
Spalding County, Georgia - Court Records
St Tammany, Louisiana - Court Records
Tazewell County, Illinois - Court Records
Vermont - Statewide Court Records
Which data sources were provided to Tyler upfront?
Complete record listings were provided to Tyler upfront for the following data sources on the dates noted:
— Butte County, California - Court Records (provided Mar 4th
— Calaveras County, California - Court Records (provided Mar 4th
— Kings County, California - Court Records (provided Mar 4th
— Mendocino County, California - Court Records (provided Mar 4th
— Merced County, California - Court Records (provided Mar 4th
— Muscogee County, California - Court Records (provided Mar 4th
— San Diego County, California - Court Records (provided Mar 4th
— San Mateo County, California - Court Records (provided Mar 4th
— Santa Cruz County, California - Court Records (provided Mar 4th
— Sonoma County, California - Court Records (provided Mar 4th
— Stanislaus County, California - Court Records (provided Mar 4th
— Sutter County, California - Court Records (provided Mar 4th
— Tehama County, California - Court Records (provided Mar 4th
— Bexar County, Texas - Court Records (provided Mar 4th
— Forsyth County, Georgia - Superior & State Court Records (provided Mar 5th
— Dallas County, Texas - Court Records (provided Mar 9th
Data sources returned by Tyler as of Apr 1st:
— Forsyth County, Georgia - Superior & State Court Records (returned Mar 9th
According to Tyler's 4/1 update here
We continue to work on securing cooperation from judyrecords.com
Considering I sent Tyler 16 data sources upfront nearly a month ago, while they've returned only 1, the above
statement is extraordinarily dishonest.
And previous updates:
judyrecords.com’s continued cooperation is extremely important
The operator of judyrecords.com has indicated a willingness to share detailed information that will assist Tyler
Additionally, my understanding now is that Tyler likely represented to California courts that I had provided data
for only 3 California courts, when in fact I had actually provided all 13 California direct access data sources on March 4th.
My jurisdiction was not in the first list. Does that mean nonpublic cases weren't exposed?
In general, no.
However, these are low risk at least as far as judyrecords is concerned.
If judyrecords has any issues with these data sources, then Tyler's entire product was access control defective, not just the most important part (i.e., case access).
As mentioned in the first question, it is my current understanding/assumption that any non classic Odyssey Portal had
the issue of no authorization check for direct access.
Depending on what the counts look like when data is provided, whether or the extent to which this is the case
will become more apparent.
Additionally, this is not considering the "any dog with a link" update issue described in the section "It Gets Worse".
How long has Tyler known about which jurisdictions were accessed which way by judyrecords?
I provided Tyler this information on March 4th, before the end of the first discussion I had with them.
How long has Tyler known about the "any dog with a link" problem?
See the section "It Gets Worse" for details.
I sent that message to the CA State Bar to forward to Tyler on March 2nd, who had been in communication with Tyler up to that point, whereas I had not.
Will judyrecords be providing any more data upfront?
Only in return at this point.
Any other form of collaboration is unacceptable.
Tyler's software had an extrordinaliy bad
defect, we've got a huge fucking mess,
and I've gone far beyond what was agreed. 3x more to be exact. See "Type 2 Collaboration And Disbelief" for details.
Yet Tyler's position has been to continue to demand more without returning data in kind.
At some point, you have to put on the brakes
You fucked up not implementing basic security, and you need to own it.
I'm not going to do anything that will hamper your incentive to collaborate back at this point,
and that means not providing data upfront, any longer. Only in return.
I have to do this because that's the only way I have a fighting chance of keeping you accountable and maximally ensuring I can get the data right on my side.
Regarding #2, Odyssey Portals (non classic) that were accessed by direct case requests only would be the scope
of cases at issue as far as judyrecords is concerned.
Also see "Tyler has said that judyrecords needs to provide full data about cases up front, and that this is necessary. Is that true?".
Were any of the unprotected case links indexed by Google from Odyssey Portals?
The number still indexed by Google as of 4/4 in fact is around a quarter million — even after Tyler changed the case URLs for all their sites.
The number for Idaho alone is sitting around 100k right now. See here
— unprotected and publicly available in Google's index from Idaho's own Tyler portal.
Google suggests the phrase "felony expungement
", seemingly related to the contents of the search.
According to Google
, cases from Idaho's portal have been indexed as far back as 2015. This is consistent with the Internet Archive's capture of the Odyssey Portal at that link here
Has Google been keeping up to date on the sealed cases of Odyssey Portal clients?
Well, yes. From Google's cache here
Sealed cases receiving updates on Google months, even years, after the fact.
Does that mean sealed cases have been publicly available from other jurisdictions on Google, like Idaho?
The number of these unprotected case links still lingering in Google's index as of today, even though it's still almost a quarter million,
is likely to be a small subset compared to what's been available over previous years.
Where was the mapping of case ID to URL provided?
The mapping was provided as part of the Odyssey search API case results object.
An actual example:
"Style": "THODAS VS FIPPS",
"FileDate": "10/7/2002 12:00:00 AM",
"Description": "Small Claims",
"DefendantName": "FIPPS, C. EDWARD",