2.1.0 Public Test
Hello, welcome to 2.1.0. This is a substantial release and I am posting it first as a non-automatic, manual update for users who want it. I'll roll it out through automatic update in the coming days unless substantial issues are reported.
Speed and accuracy improvements
2.0.3 introduced the concept of a 'cloud save manifest', an approximation of an actual Steam Cloud behavior. 2.1.0 does stuff with it.
To start, upload and download time is substantially reduced. Client UI latency is reduced. Accuracy to real Steam Cloud behavior is hugely improved in 2.1.0. Most of these improvements are due to having this new system. Accuracy is better because I can flow through the real Steam Cloud system more of the time.
Steam Cloud itself is based on a bunch of....Microsoft technologies. It's a database of sorts. It's atomic. Google Drive is not atomic. OneDrive is, kind of, but it lacks file versioning (which Google Drive does have)....but neither are a database. Steam Cloud handles partial writes in a way that is perfect by definition.
We now approximate atomicity. A single state.cloudredirect replaces separate cn.cloudredirect & manifest.cloudredirect. A consequence of introducing quasi-atomicity is a less 'pleasant to look at' folder structure on your Cloud remote. Let's say your game save is skyrim.sav.
The old folder structure would be CloudRedirect/<SteamID>/<AppID>/Blobs/<Game Save Folder Name>/skyrim.sav. The new structure is CloudRedirect/<SteamID>/<AppID>/Blobs/<Game Save Folder Name>/skyrim.sav/<SHA1>, with the file with the SHA1 name being the skyrim.sav itself. In a failure scenario where the Steam Client is not downloading saves or something along those lines, everything in the cloud provider end is still human readable and human recoverable. You can still manually retrieve files from a cloud provider if you need to, you'll just have to rename them to the name of the folder that they are inside of.
Anyway, that failure scenario is pretty uncommon and the solution is self-evident. The failure scenario involving bad networks was more common and it is solved in the native, correct way.
What I describe above was a pretty annoying thing to implement, but we have effectively full atomic read/write. This prevents a scenario where the cloud data is lost during a crash/bad network connection during an upload/etc. While the local machine would retain the data it had, Steam would throw sync errors and stuff would get out of whack. Fixed.
More Manifest Stuff
Steam Cloud's 'save manifest' delivers a bunch of information to the local machine, information Steam uses to figure out what to grab/not grab from the cloud. CloudRedirect now collects that information, puts it in the cloud, pulls that information on demand and provides it to Steam when Steam wants it. Per file SHA, timestamp, size, persist_state, platforms_to_sync, blah blah get handed over and Steam makes decisions. This helps speed, not accuracy - I've long had a hack that achieved the same accuracy but with a performance penalty. That hack now is dead.
Multi-machine session lock
Steam has a thing to prevent a scenario where a user starts playing on PC 1, PC 1 goes offline while still in game, user starts playing the same game on PC 2 while online, saves, quits, PC 1 comes back online, user saves and quits.
There is a system to prevent the data loss you can see coming in that scenario. CloudRedirect now has that exact system, wired up how Valve uses it for native Steam Cloud. Because we flow through the normal process now. The user will get the conflict dialog, as intended. Though this is the definition of an edge case, it is here.
Save quota and rules injection
Steam Cloud is not an all you can eat buffet. Many games have maximum save file quotas, set by the developers. They have save file patterns that must be met in order for saves to upload. We enforce those now. This prevents files that should not be in a game's save folder from being unintentionally uploaded, solving for specific dumb SteamTools related problems while simply being more 'native'.
We inject the real rules for each application. Save scheme, excluded directories, excluded files. Rather than parsing it outside of the real process and then allowing sync on files that meet the spec, this process is native now. Native means me not having to chase down sync errors.
Linux, too!
The Linux version now requires less per-Steam Update work on my part. Lots of improvements to the Linux variant for better distro support and better thread safety. The install script returns! Run curl -fsSL headcrab.pages.dev | bash and select CloudRedirect and you'll get up to date with the latest version. The Linux version still requires games be set as AdditionalApps in the SLS config & the DisableCloud=No setting. Works well in testing.
Grab bag:
Speed improvements for cloud providers by eliminating redundant calls, Google Drive duplicate file detection and cleanup, auth failure dialog shown once per session, some extra minor features that are Windows-specific. Lots. What is new is 'lots'.
Coming up next:
2.1.0 is the final large release in a series of them. What remains in terms of workload is small.
There are still two (arguably one) AutoCloud functions that are handled by CloudRedirect and not by the native Steam Cloud system. The work to fix that is done and is ready to start internal testing. Just a win for maintainability and accuracy but probably not something anyone will notice. This will be a small point release, let's say 2.1.1, with the actual numbering (of course) subject to change as things come up that require fixes before we get to that point.
I'll start moving things that are of absolutely no consequence to the native system, too. Stuff like the 'arrow' that indicates the direction of cloud sync - download or upload - will flow through the native system. Same timeline as above. With both of these outstanding issues made native, I will finally be happy with the state of this tool.
I need to add in a way for users to migrate from one cloud provider to another. Currently that is a manual process and is fairly fraught. Probably 2.1.5.
I'll do a minor update to address one scenario, which is Linux specific: User is offline. User plays game, saves, quit. User gets sync error nag! Because, yeah, we can't sync. But that's because we're offline, duh. Simple fix, but it didn't make the cut for this release. This release cycle was brutal and something needed to give. 2.1.1.
Windows version may get game update disabling, an easier to use version of manifest pinning. Works the same way internally, just easier to use for an end user. Probably 2.1.5.
2.2.0 will introduce a fully rewritten achievement/playtime sync system and fix the Steamtools issue where some games do not 'show' achievements. Work has begun on that and is going very well. Playtime sync works correctly in an internal build. This new functionality will be available for both Linux and Windows.
2.1.5 is a smaller release. It will add support for some non-SteamTools Windows unlock solutions, focusing on ones that are not transparent clones of Greenluma and either have an actual user base or do things that are interesting to me. No promises on specific client support, though.
What I can promise is a way to finally delete the junk Steamtools put in your actual Steam Cloud, under appid 760. That'll be in 2.1.5, available as an option.
But first, some time off. I'll take a break for a couple of days before getting started on any of this. I'll address breakage and stuff like that if it develops, but this was a lot. Some rest is genuinely needed.
Known issues:
Switching cloud providers in a scenario were both the new and the old provider have different CloudRedirect data for the same user/game can cause data loss on the remote, though that loss is recoverable if the remote is Google Drive. The local on disk save will overtake the data on the remote unless the user moves the local save and the local remotecache.vdf out of their folders for the given game before switching providers.
In other words: if you swap from OneDrive to Google Drive with different sets of progress for the same game on the two of them, expect a bad time.
I don't really consider this an issue as it is a behavior Steam Cloud itself would never make possible, but it does exist. I plan to mitigate by checking the remote for this when the user goes to change providers in the UI. 2.1.1, I think.