Jump to content
Sign in to follow this  
ThisIsMe

Oblivion and BSAs - and why I use them

Recommended Posts

I'm fairly passionate about keeping game resources in archives (BA2s, BSAs for Bethesda games), for either the matter of load speed, avoiding fragmentation or simply general tidiness.

I have no knowledge of any modern AAA game that doesn't respect this guideline either.

Anyway, I've read several topics on the matter with great interest, on these forums and elsewhere, and while it appears that archives are definitely the way to go for recent games (SSE, FO4), there is no such consensus for Oblivion (yet).

I recently wondered why every one of the UOP of Bethesda games came as BSA files, but the Oblivion one. Someone on the team had the kindness to answer me, and my curiosity was somewhat sated.

Yet, I still am curious to know whether packaging files, especially those nasty, tiny and numerous voices/lip-sync files, would cause any serious harm to my game.

Instead of clogging the official Nexus page with my questions, I came here directly, to the experts and the authors of the mod, to gather further insight.

You see, the advantages of keeping stuff packed are many to me :

  1. I have yet to use a mod manager that doesn't load mods packed with BSAs faster than loose files. Since I spend arguably more time modding than playing, this is an advantage to me.
  2. I installed the Oblivion base game on a SSD, and have my mod files separately on a HDD. While I haven't done any objective tests, or moved all my mod files to a SSD as well, it's my gut feeling that past a certain point in a heavily modded Oblivion game, the game seems to load faster and definitely runs smoother for me than with loose files. My HDD, albeit recent and of a good brand, just doesn't seem to suffer as much when I monitor it.
  3. Most mods that were created for Oblivion are old by now, and created with tools or methods that could be considered as unsatisfying by modern standards. It's not unusual to find wrongly named resources, duplicate files or rogue files, like thumbs.db or desktop.ini, delivered with them. Using BSAs, I simply don't have to care about these issues, since archivers (like BSArch) refuse to acknowledge such bugged files. My game is therefore cleaner than it might have been without BSAs.

Finally, there is at least one example of official Oblivion BSAs colliding, between DLCKnights and DLCFrostCrag, for the following resources :

  • /sound/fx/ambient/amb_tower_lp_01.wav
  • /sound/fx/ambient/amb_tower_lp_02.wav
  • /sound/fx/ambient/amb_tower_lp_03.wav

So, in the light of the answer that was given to me about why the UOP chose to keep loose files instead of going with archives, my question is the following :

  1. if I made sure that my BSAs always have proper and unique timestamps, in accordance with load order (thanks to any decent modmanager)
  2. if I respected proper compression standards (none for sound and voice files, for example)
  3. if I didn't care about *some* occurrences of mismatching resources (wrong meshes, textures flickering ?), because the engine could somehow get confused between loading them from two different BSAs
  4. if I ran most up-to-date game-stabilizing tools, like GameEngineFixes, NVAC, etc.

would I run into severe issues, like risking a crash, directly because of my choice to package content into BSA, only when it makes sense of course, rather than keeping said content as loose files ?

Consistent game crashes would indeed be a show stopper for me, but the only one. Given the few reliable Oblivion debugging tools at my hand, I'd prefer to avoid running into them rather than having to debug them. On the other hand, with the recent improvement of MO2 which is now able to display conflicts even between files packed into BSAs, it would be a shame not to further use the facility that BSA have been in my modding adventures so far.

I'd really appreciate your insight on this. Thank you very much in advance for your time.

Share this post


Link to post
Share on other sites

You will run into issues if you attempt this with non-unique assets. The game will either load them at random, meaning you won't see the changes you intended to see most of the time, or you'll just get a pile of CTDs. Since it can't be predicted when a file collision will just load wrong or become a CTD it just isn't going to work to do this. That applies even to the official archives and is likely one reason the game isn't entirely stable. Your only recourse there would be to unpack both of them and run them loose.

That's why the consensus in the Oblivion community settled on loose file installs long ago. The game engine simply won't tolerate the collisions. That's why UOP ships loose. That's why pretty much everyone else does too. BSAs are only useful when dealing with unique assets that don't share filenames in other mods. That's why you don't see them anywhere near as often.

Share this post


Link to post
Share on other sites

I understand, thank you Arthmoor.

This is a shame really, but I certainly won't contest the experience that your team has gathered over the years. It might even explain some of the random CTDs that I experienced in the past.

Also makes me want to reconsider the interest of mods like :

which have way too many loose files for the "value" that they offer compared to better load speed.

Anyway, I guess the only option I have left then is to gather everything, identify collisions, keep those collisions as loose files, and extract the unique resources, to repackage them as BSAs.

That should work, right ?

Oblivion has to be the most knowledge demanding game among all those that I have modded : Skyrim, SSE, FO4, FNV, and even Morrowind (because it has no xEdit and no Nexus to speak of).

Share this post


Link to post
Share on other sites

FO3 and FNV also had the same problem as Oblivion with BSAs overriding, however turned out that the engine supports this just fine as in Skyrim and later games afaik, this feature is simply disabled by default and has to be enabled via script extender plugin.

So your best bet is to contact any remaining active programmers who develop OBSE plugins and ask to check that up, maybe it is the same case here.

Share this post


Link to post
Share on other sites

Thank you very much for chiming in, Zilav.

What a beautiful way it would be to conclude the Oblivion chapter of the UOPs if the community could achieve that feature.

As I said, I've modded all the major (moddable) Bethesda games by now, and Oblivion has to be the one with the most quirks, probably because it's the game in which the engine really took off as a modding tool to the masses without being quite ready for it. No offence to Morrowind, which I still hold dear to my heart, but I still remember the day where we got official support to add OBSE capabilities to the Steam exe, and that was far from being acquired at game launch.

Everything was done, and learned along the way. What kind of textures should be used, in which format and for what purpose, what kind of meshes best to be pyFFI'ed, what kind of tweaks to be added to the ini, how to build a proper LOD, how to use nascent ENB/OBGE shader support or tottering levelled lists injectors, how to get rid of the 200+ hours animations bug. Everything picked up from there, every other game benefited from the experience gathered on Oblivion, even Morrowind, even the Fallout games.

Nowadays in Skyrim, you just plug in BethINI, pick an ENB, run xEdit/xLODGen, Merge Plugins/zEdit, install the UOPs, etc., and you're good to go. You don't have to question anything, it just works. CTDs are issues 95% of the time caused by user management, and much rarer thanks to a more forgiving game engine.

Because of these quirks that Bethesda never cared to iron out, Oblivion is much less straightforward to mod than other games. CTDs happen at the most random and the weirdest places, despite an almost anatomical care, which is oh, so frustrating. Yet, it's a game that deserves to be modded fully, be it because there are some REALLY outstanding mods or tools that people shy down from using, or be it for the fact alone that Cyrodiil is the centre piece of the whole continent and deserves to be properly represented.

I will be honest with you : I have been mostly lurking throughout the years, learning from where I could, contributing whenever I could, but I never got anywhere close to any of the OBSE team members to see myself in position to ask anything of them today. Of course I know of the official site, and I will petition there if there is no other way, but hopefully I can use this thread to also get more attention and especially support from people who KNOW what's really happening, much better than I ever could explain it to a probably over-scheduled dev who needs precise, clear and down-to-earth instructions that can get us from point A to B, no less, no more.

As you can see, I registered to this site just yesterday, because I thought that I was just not of the level of knowledge required by Oblivion. I do contribute much more to Skyrim or Fallout 4 reddits or discords, because their engine is just really much more transparent. I thought I started to know stuff about Oblivion, but the truth is that if you haven't got the years long experience about how its engine works from behind the scenes, you probably don't know much about it, and even then. Yesterday's "event" just proves it even further.

I won't make another tirade about how BSAs really can help in getting mods out there, in a proper and efficient format, for the aspiring modders to consume. As of now, there are many well-known mods of otherwise high quality on the Nexus delivered with BSAs which do collide between themselves. No, I won't name them, but they're easy to spot in any heavy load order. Hell, there are even official BSAs colliding. If we acknowledge the existence of this undesired root of CTDs, then something should be done about it.

So my conclusion will be obvious : what chances are there for this (in my opinion essential) OBSE tweak to happen ? ShadeMe to the rescue ? What can I (as a non-programmer) reasonably do, because I will definitely try to give it my best shot ?

Share this post


Link to post
Share on other sites

If all you're really worried about is disk IO performance with the game then I wouldn't spend a lot of time on the subject. The only test I've ever seen where BSA vs loose file mattered at all was when someone decided to unpack the entire vanilla archive set and run the game entirely on loose files. The degradation was noticed, but wasn't significant enough to be of concern. If you're on an SSD I doubt even this test would make a dent now.

You should also keep in mind that even if this DLL that zilav is talking about turns out to be the cure to the problem and BSAs could become a thing for every Oblivion mod, we won't be doing this with the UOP because that DLL would place a hard requirement to use OBSE on the project. Right now it has a soft requirement for a couple of minor fixes but that's it.

Share this post


Link to post
Share on other sites

I'd love to see some hard data on speed differences with loose versus BSA'd resources.  There has been an article or two about BSA compression as well; does it improve performance to be compressed or not?  A more detailed study on that would be great.

Basically for Oblivion, BSAs are a tidiness thing.  If the assets of the mod are completely new and unique, a BSA can be a prettiness tool.  If we had data on compression and it's performance impact, we could say BSA's potentially improve performance.

The Bethesda modding community really took off with Skyrim.  That's when the research for every damn question about the game took place.  Modding wasn't quite as popular with Oblivion.  Nexus blew the hell up with Skyrim.

Share this post


Link to post
Share on other sites

@Arthmoor

I am not surprised that you come back to question my usage of "efficient", but performance at this point is only a secondary concern of mine.

I really meant "efficient" as in "ease of install/uninstall" and "able to unequivocally scratch out causes of fatal installation errors".

If you deliver a mod with a BSA, short of extracting and messing around with it, there is just no way that the average user with a decent mod manager could cause involuntary issues, contrary to loose files which can be packaged :

  • for the first within a /data/meshes/whatever folder
  • for the second within a /meshes/whatever folder
  • for others still with no folder at all, with custom folders, etc.

The game knows what to do with the content of BSAs, what should go in them, the inexperienced modder does not. In that way, BSAs are a much more efficient and reliable way for distribution of large and popular mods, yes.

Also, before someone questions me about that, I have nothing against loose files. Like BSAs, there is just a time and place where their use should be promoted because it's optimal : for smaller mods, for patches, for updates, etc. However, I do think it's unreasonable to expect users to have to extract their whole base game folder, and then all the major game mods in their load order, because that's currently what we ought to do to get a chance at escaping *one* potential cause of CTDs on Oblivion. This workflow would be utterly cumbersome to maintain, and detrimental to any serious attempt at modding Oblivion beyond some simple graphical setups like Bevilex's. Here we could indeed talk about performance advantages of BSAs, since all known mod managers literally choke on loose files past a certain boundary.

About the Oblivion UOP, you'll find me among the last to demand anything, or criticize the work of your team. My goal is to raise awareness, support, and hopefully find a solution to an issue that plagues the Oblivion modding community which as far as I can tell remains mostly unaware of it : no warning anywhere on the Nexus pages of mods coming with BSAs prior to downloading and installing them, for example.

Quite like the UOP, and then engine stabilization tweaks (NVAC or equivalent), and then engine fixes, the aim instead is to eliminate step-by-step what seems to be another cause of a severe engine issue, to achieve as stable a modded game as possible with what we currently know, for the enjoyment of the modders and the users of the produced mods.

Share this post


Link to post
Share on other sites

You have to keep in mind that the Oblivion community existed before abominations like MO, NMM, or Vortex existed. We all used either Wrye Bash (preferred) or OBMM (the predecessor to everything Nexus is doing today).

With OBMM it was indeed possible to make a giant mess of things because the same terrible design principles NMM/Vortex use today came directly from OBMM. Wrye Bash is by far the preferred solution for installing mods because it has a very robust tracking system that's not going to just arbitrarily screw up your game.

I was not suggesting anyone extract their entire vanilla BSA set. I was pointing out that the testing that was done back in the day found that that was the length you'd have to go to to degrade performance enough to notice it while playing. To be perfectly blunt, I find the practice of unpacking BSAs to be moronic in Skyrim/FO4 and there being a mod manager that specifically catered to this has been nothing but a damaging influence instead of being a helpful tool to advance the cause.

@Malonn you vastly underestimate how much Oblivion mod was a thing. In its prime the community for that game was easily as large as Skyrim is now. Activity surrounding the game was largely centered on the old Bethesda forums though so Nexus just didn't seem to be a big thing. There's a reason Oblivion remains the #2 game on Nexus for the raw number of mods. :P As far as the testing data, it's buried somewhere on the archived forum. It was quite extensive and very detailed and proved that BSA use would not be a significant factor in anything for those mods that could even use them. In theory they should improve things, but in practice that improvement was too small to measure in game terms. So while it would indeed be much tidier to use BSAs, the engine defects in the game just don't make it possible. Especially for something like the UOP.

Share this post


Link to post
Share on other sites
1 hour ago, Arthmoor said:

@Malonn you vastly underestimate how much Oblivion mod was a thing. In its prime the community for that game was easily as large as Skyrim is now. Activity surrounding the game was largely centered on the old Bethesda forums though so Nexus just didn't seem to be a big thing. There's a reason Oblivion remains the #2 game on Nexus for the raw number of mods. :P As far as the testing data, it's buried somewhere on the archived forum. It was quite extensive and very detailed and proved that BSA use would not be a significant factor in anything for those mods that could even use them. In theory they should improve things, but in practice that improvement was too small to measure in game terms. So while it would indeed be much tidier to use BSAs, the engine defects in the game just don't make it possible. Especially for something like the UOP.

I remember.  The official forums were the place to hang out and chat about Oblivion and mods.  Releases, WIP's, etc. all in the official forums.  I didn't know Oblivion has the second highest number of mods in all of Nexus.  Let me guess: Skyrim is #1?  Okay, maybe Oblivion had nearly as large an active community, but for sheer numbers of players with modded setups, Skyrim reigns.  Nexus officials said themselves that it was when Skyrim came along that they had to really improve their backend to cope with the traffic.  I remember issues in Oblvion's day too.  Doesn't FO3, FO4 and FNV and SSE have high mod numbers on their servers?

Share this post


Link to post
Share on other sites

Right now the top 5 games in order of the number of mods uploaded is: Skyrim LE, Oblivion, Fallout 4, New Vegas, and Skyrim SE. Fallout 3 and Morrowind are 6 and 7 and after that each game drops substantially in the number of available mods.

Yes, Skyrim made them rethink how they were doing some things there but that doesn't in any way diminish how strong the Oblivion community was up until around 2012. Oblivion modding didn't really drop significantly until after the Skyrim CK finally came out.

Share this post


Link to post
Share on other sites

Gotcha.  I guess my only data to back my opinion is following the issues Nexus had after Skyrim (and then the CK) were released.  Plus you look at the downloads for some Skyrim mods and you're looking at the millions easily (unique).  If an Oblivion mod is in the 10K range (unique) it's popular (nowadays), but 10K for Skyrim is near-unused.

But I've been with Oblivion since '06 so I guess I should have known better.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

Support us on Patreon!

×