Community Package Repository Manage Chocolatey Packages through Boxstarter
Resolved - The majority of site changes have been completed and the site is now stable. This has been a very long recorded outage however, while the site has been stable and working for some time we have had to make additional changes to make this stability more permanent. There are some additional changes required and you can find these on GitHub. Thank you for your patience. (11:04 — Dec 13)
Monitoring - Site maintenance is now complete. (16:17 — Dec 11)
Maintenance - We have taken the site down for a short maintenance period and will update the status once complete. (15:52 — Dec 11)
Monitoring - It was determined that the intermittent failures has to do with package dependencies that have a `.` in their name. Possibly also packages with a `-`. So we’ve backed out the change until we can get the index to respond properly on those types of requests as well. (23:10 — Dec 8)
Monitoring - We are receiving a few reports of folks who are receiving the unable to resolve dependency again - right now it is intermittent, but we will continue to monitor the situation closely. (22:50 — Dec 8)
Monitoring - The site started seeing high latency again, so we deployed an update to use the index cache for FindPackagesById, which removed the latency issues. We’ve taken the affected systems down to just the community repository when using the packaging services. (18:37 — Dec 8)
Monitoring - A number of users have reported issues with dependent packages not being found when they are installing a package. A fix has been been implemented and we are continuing to monitor the issue. (18:29 — Dec 6)
Monitoring - We have implemented some changes to the caching for the site (see 761 and 762) which should minimise the issues experienced when searching. We are monitoring the effect this has on the ‘OutOfMemoryErrors’ the site has been experiencing. (10:50 — Dec 6)
Investigating - Unfortunately increasing the resources to the web machines has not resolved the issue and users are continuing to see ‘Failed to process request. ‘This package had an issue pushing: this writer hit an OutOfMemoryError; cannot commit’. Restarting the web machines alleviates the issue temporarily however we are continuing to work with our hosting provider for a permanent resolution. (10:41 — Dec 5)
Investigating - We are still working on a resolution for the caching changes and stabilizing that aspect to allow for the numbers to be correct. We increased the resources on the web machines to the maximum available with our current hosting provider and are no longer seeing memory issues, but there are still intermittent latency issues. (01:45 — Dec 4)
Investigating - After the implementation of additional caching we are investigating issues of inconsistent package totals shown on the packages home page. We are also investigating issues around package status updated being made from the package page itself (this appears to be limited to package maintainers and community moderators). We apologise for the continued issues and assure you we are continuing to work on them but with the goal of minimising downtime. (14:18 — Dec 3)
Monitoring - We have turned back on the aggregate stats with a more performant implementation. We’ve also adjusted some database column sizes and added a missing index to increase performance. We still have a few more things to do, and we are not ready to adjust the severity down any further until those aspects are implemented. (05:15 — Dec 1)
Monitoring - We have implemented changes to stabilise the site and are monitoring the issues. In the meantime we have changed the Severity from Down to Disrupted. (21:45 — Nov 30)
Investigating - We’ve determined there are some queries that are queueing up and timing out and are working on a resolution. We’ve also shut off aggregate stats temporarily so everything at https://chocolatey.org/stats will come back zero. (19:45 — Nov 30)
Investigating - We’re aware of users experiencing slow and intermittent connections to the website and community package repository and we are currently investigating. We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information. (22:38 — Nov 29)