Update on EFM slowness and instability issues. Tuesday, 8/16.

Early this morning, Tyler Technologies (the State's E-Filing vendor) released a new version of their EFM*. 

This release included new logging tools that they hope will allow them to diagnose the slowdown that occurred during the last EFM upgrade on July 22nd.

Unfortunately, this current upgrade has, so far, proved to be both slower and less stable than previous verions. That is disappointing. For now, we are urging patience while Tyler works to solve the problems with this latest upgrade. They are fully aware that it's a mess and are actively working to resolve the new instability issues.

As I'm writing this post, the EFM is down, with no ETA for a fix. We hope they'll get it fixed up before the end of the day, but if you miss a filing deadline due to this instability, please refer to our previous post on how Indiana's trial rules protect you in that situation.

This is frustrating. We are now approaching one month with the EFM running at roughly 1/3 of its previous speed, and know the situation is hard on you as well. We continue to cooperate with the State and Tyler to provide the information they need for diagnosis. We also remain committed to pursuing this until it gets fixed. 

* EFM = E-Filing Manager, the system through which all filings are routed to destination courts.

Friday update on the E-Filing slowness issue.

Yesterday, (8/4) we participated in a discussion of this problem with the State's EFM* vendor (Tyler Technologies.) A few important notes from that meeting:

1) The network routing and latency issues that we had hoped might provide an explanation have been eliminated as a cause.

2) The slow response time is affecting all Indiana filers, regardless of their EFSP**.

3) More diagnostic information is needed, so early next week, Tyler will be deploying a version of the EFM that logs more information in support of the diagnostic effort.

4) The result of #3 is that we know this won't be fixed in a matter of days. It will be at least a week while the new logging software is deployed. (*sigh*)

We came away from this meeting frustrated, but with the understanding that Tyler is working diligently on this problem. They are fully aware of the pain it is causing filers, particularly those of you who submit high volumes of filings where response time is critical. We will continue to support them in their efforts by providing regular feedback from our own logging system.

* EFM = E-Filing Manager, the system through which all filings are routed to destination courts.

** EFSPs = E-Filing Service Providers, The web sites used by filers to compose and submit filing envelopes.

Tuesday update on slowness issues. Still making small incremental gains. Current hypothesis: Network routing may play a role.

 Over the weekend, Tyler Technologies shut down their system briefly to do some performance tuning on the EFM API. Now that we've got a full day of logs to examine since the most recent change, we can report that there has been a 16% decrease in the average response time compared to last Monday.😊

That's good news, and worth celebrating, but that's still over 3 times the average response time prior to their cloud migration, so this is still a top priority for us.

We've been a bit frustrated by the fact that it was a change to someone else's system that caused this. That means we're in the awkward position of just measuring slow response times and complaining instead of being able to contribute to a solution. However, one of our operations folks pointed out that network latency is something we can assess from our side of the connection, so we've been doing that and are finding some definite problems that we can use to start some diagnosis from our side of the connection. We're working that angle now, and have passed along our findings to Tyler in the hope that it will help.

Weekend update on the slowness issue- Marginal gains on Friday.

We are still not seeing anything like a return to the original performance for the E-Filing Manager API.

Some changes were made by the State's vendor early Friday afternoon that appeared to result in a small improvement, but it's hard to tell for sure because Friday afternoon is a time when our load is usually lighter than the rest of the week.

We're hoping the small gain is still discernable on Monday. However, even if this gain is solid, we acknowledge that it was small and we are not even close to calling the issue resolved. There's still far to go.

Thursday Morning update on performance issues. Bottom line: No noticeable improvements.

Early this morning (Thursday, 7/28) Tyler Technologies took the State's E-Filing Manager (EFM) off-line to do some performance tuning. We have been watching closely this morning, and as of 9AM, it looks like no improvement resulted from this work. (sigh.)

We'll be reporting these results to Tyler Technologies shortly so they can get to work on next steps.

Note that at 9AM, our sample size is fairly small, so our conclusions may be adjusted as the day unfolds, but so far, we're calling this round of tuning ineffective.

E-filing will be offline at Midnight while Tyler Technologies works on performance tuning.

We have been notified that Indiana's E-Filing Manager (EFM) will be offline for roughly one hour starting at midnight tonight (7/27/22).

That's good to hear, because as you doubtless know, the recent migration of the EFM's servers to the cloud went very poorly, and we've all been struggling with an extremely slow system this week.

Given the results of previous tries at performance tuning, we're not going to do the happy-dance yet, but we're hopeful and will wait to see what the morning brings.

If you'd like more background on this saga, take a look at our previous three posts on this topic.

Update on slowness isues with the EFM as of 1PM Tuesday, 7/26

For what has transpired up to this point, take a look at our post from last night about what caused this issue.

Here's the new information from today:

Yesterday, the State's EFM was running so slowly that it was effectively unusable. We estimate that the EFM was taking 17 times as long as usual to respond to the EFSPs such as Doxpop.

Today, it started off looking good, but a little slow. However, as of Noon today, our estimate is that the EFM is still taking 7 times as log to respond as prior to the cloud upgrade.

We have passed this information along to Tyler Technologies along with an urgent request to essentially double whatever change they made last night. That change got them halfway to the goal, so we expect it should not be hard to travel the rest of the way.

We share your frustration with this extended problem. Please know that we're doing everything we can, and at this point we're waiting on the folks at Tyler to respond.

The E-Filing outage on Friday and Monday- What happened?

 To understand this explanation, you need to know that there are two distinct sections of the E-Filing system in Indiana:

1) The section you interact with as a filer is called an E-Filing Service Provider (EFSP). The largest EFSPs in Indiana are the Tyler system, the Indiana Prosecutors' CMS, The Public Defenders' CMS and Doxpop. We provide the part of the system you interact with to get your work done.

2) Behind the scenes, all of the EFSPs connect to a single E-Filing Manager (EFM). The EFM takes the filing information that the EFSPs assemble into envelopes and write that information to the Odyssey system used by the courts. Tyler Technologies operates Indiana's EFM.

So in our case, the primary flow of data is Doxpop > EFM > Courts.

On Thursday evening, Tyler migrated Indiana's EFM to a cloud-based server. It didn't go smoothly. The result was a nearly complete outage lasting from Thursday night through the close of business on Monday. During this time, the EFM was extremely slow- Slow enough that it was essentially unusable. 

On Friday, all EFSPs were affected. Over the weekend, Tyler managed to get the connection to their EFSP working well. However, the interface used by the Prosecutors, Public Defenders, Doxpop, and other smaller EFSPs was still unusable on Monday. Doxpop reported this early Monday morning, and provided Tyler with all of the information we had available.

At 9:50 PM on Monday, Tyler reported that they completely fixed the interface used by the EFSPs, including Doxpop, that were affected on Monday. We'll be monitoring the system closely on Tuesday morning to confirm this, and ask that our users quickly report any issues they run into so we can let Tyler know if the fix applied Monday evening did the job.

Thanks to all of our users for your extraordinary patience and kindness during Friday and Monday. We were having two very bad days, but you made it a lot better by being so understanding!

Similarly, although it's tempting to point fingers, we really can't complain about the team at Tyler. Upgrades of this magnitude are no fun for anyone. They were having a very bad day too, but Tyler stuck with it until all of the issues were resolved. We appreciate that their technicians were hard at work well into the evening tonight.

Did this outage cause you to miss any filing deadlines?  If so, please read our previous blog post, which explains how the trial rules address this situation.

Did the e-file outage on Friday through Monday cause you to miss a deadline?

On Friday, July 22nd, we noticed an increasing issue with slow or completely unresponsive response times being returned by the State's E-Filing Manager, which is the central system that routes E-Filing submissions to each court. 

Doxpop promptly reported the issue to the State's vendor for this system, and received confirmation they were "aware of an issue which is affecting filings in IN Prod and are currently investigating".

As of 5PM, this problem was still not resolved, and the State's vendor indicated that they would continue working on the problem into the weekend.

If these issues caused you to miss a deadline, please note that Trial Rule 87(J)(1)(c), essentially provides for a one-day extension when a lengthy e-filing system outage or issue prevents filers from meeting a deadline.

We expect that the Trial Courts will all be aware of the scope and timing of this outage via direct communication from Trial Court Technology. However, if the court you are working with requires an affidavit from Doxpop to attach as an exhibit to the motion described in TR 86(J)(1)(c), please let us know and we'll be happy to supply documentation in support of your motion. (Simply attaching a screen print of this blog post may be sufficient.)

On the appellate side, take a look at Rule 68(M)(1)(d), which appears to be nearly identical to the Trial Court rule on this topic.

USPS Postal Rate set to increase, how will this effect Delivery by Doxpop rates?

 

As announced by the USPS earlier this year, USPS postal rates are set to increase on July 10th, 2022. We've reviewed those changes and will be updating the pricing to our Delivery by Doxpop services in line with that pricing at the end of the day on July 8th due to the update happening over the weekend.

The rate updates are detailed but in short, the only increase you will see is the actual postage increase for services that utilize the USPS. The only exception is with First Class Mail and Certified Mail flat packages. With those, you will see the per-page printing fee slightly increase as that is how we account for additional per-ounce charges. 

You can review our full Delivery by Doxpop fees on our fee schedule here. 

If you have any questions or would like more details on these updates please don't hesitate to contact Doxpop Support at 866-369-7671 or support@doxpop.com.