Jump to content
Arthmoor

Plugin Files And You (ESM/ESL/ESP)

Recommended Posts

@SQr The latest is always on second post here

although the fixes  I refer to are in the latest python code - will pack a standalone asap:

307.201711250133

 

20 hours ago, WrinklyNinja said:

I've now updated my stuff to treat all .esm and .esl files as having the master flag set, even if it isn't, and I also check for the light master flag as well as the .esl extension (so a .esp with the light master flag set is treated as a light master).

On my part I never look at the esl flag - I had load order in mind and tests used to indicate that it only takes into account the extension. So an esl-flagged esp will be treated as an esp, re: load order. I do nothing new with the master flag, just what I used to do - differentiate between esp and esms

Share this post


Link to post
Share on other sites

With todays updates the .ccc files include more new cc plugin names

Skyrim SE

ccBGSSSE002-ExoticArrows.esl
ccBGSSSE003-Zombies.esl
ccBGSSSE004-RuinsEdge.esl
ccBGSSSE006-StendarsHammer.esl
ccBGSSSE007-Chrysamere.esl
ccBGSSSE010-PetDwarvenArmoredMudcrab.esl
ccBGSSSE014-SpellPack01.esl
ccBGSSSE019-StaffofSheogorath.esl
ccBGSSSE021-LordsMail.esl
ccMTYSSE001-KnightsoftheNine.esl
ccQDRSSE001-SurvivalMode.esl
ccTWBSSE001-PuzzleDungeon.esm
ccEEJSSE001-Hstead.esl
ccQDRSSE002-Firewood.esl   <-- New

 

Falloout 4

ccBGSFO4001-PipBoy(Black).esl
ccBGSFO4002-PipBoy(Blue).esl
ccBGSFO4003-PipBoy(Camo01).esl
ccBGSFO4004-PipBoy(Camo02).esl
ccBGSFO4006-PipBoy(Chrome).esl
ccBGSFO4012-PipBoy(Red).esl
ccBGSFO4014-PipBoy(White).esl
ccBGSFO4005-BlueCamo.esl  <-- New
ccBGSFO4016-Prey.esl
ccBGSFO4017-Mauler.esl
ccBGSFO4018-GaussRiflePrototype.esl
ccBGSFO4019-ChineseStealthArmor.esl
ccBGSFO4020-PowerArmorSkin(Black).esl
ccBGSFO4022-PowerArmorSkin(Camo01).esl
ccBGSFO4023-PowerArmorSkin(Camo02).esl
ccBGSFO4025-PowerArmorSkin(Chrome).esl
ccBGSFO4033-PowerArmorSkinWhite.esl  <-- New
ccBGSFO4024-PACamo03.esl  <-- New

ccBGSFO4038-HorseArmor.esl
ccBGSFO4041-DoomMarineArmor.esl
ccBGSFO4042-BFG.esl
ccBGSFO4043-DoomChainsaw.esl
ccBGSFO4044-HellfirePowerArmor.esl
ccFSVFO4001-ModularMilitaryBackpack.esl
ccFSVFO4002-MidCenturyModern.esl
ccFRSFO4001-HandmadeShotgun.esl
ccEEJFO4001-DecorationPack.esl
ccGCAFO4003-FactionWS03BOS.esl  <-- New
ccGCAFO4005-FactionWS05HRPink.esl  <-- New
ccGCAFO4006-FactionWS06HRShark.esl  <-- New
ccGCAFO4008-FactionWS08Inst.esl  <-- New
ccGCAFO4009-FactionWS09MM.esl  <-- New
ccGCAFO4010-FactionWS10RR.esl  <-- New
ccGCAFO4007-FactionWS07HRFlames.esl  <-- New

Share this post


Link to post
Share on other sites

Spotted a section on UESP about the form ID structure for ESL files. For anyone who wondered about the 4096 form limitation their explanation of how the IDs get mapped should prove interesting.

https://en.uesp.net/wiki/Skyrim:Form_ID#Creation_Club

For those who prefer the short version though, only the last 3 digits get used for a unique ID, and FFF = 4095 which is where the limitation comes from.

Share this post


Link to post
Share on other sites

I added the ESL flag to a ESP that was adding a tree and disabling several other trees from another ESP. Its master was a normal ESP using E2 records. After the flag was added the trees were still properly removed, and the other plugin was still using E2 records. The only apparent change was that the ESL flagged plugin was using FE001 instead of E3. I tested this with the same save each time (One where the removed trees were still present)

Share this post


Link to post
Share on other sites
On 6/13/2018 at 6:27 AM, jss2a98aj said:

I added the ESL flag to a ESP that was adding a tree and disabling several other trees from another ESP. Its master was a normal ESP using E2 records. After the flag was added the trees were still properly removed, and the other plugin was still using E2 records. The only apparent change was that the ESL flagged plugin was using FE001 instead of E3. I tested this with the same save each time (One where the removed trees were still present)

That's interesting.

I just did a little test of my own. The mod I used was Faction Crossbows. I copied the record for the Iron Crossbow as an override into two plugins. The first one I set the damage to 16, and flagged it as an ESL in xEdit. The second one I set the damage to 12, and kept it as a normal ESP. I set the load order to:

Faction Crossbows.esp (10 dmg)

test1.esp (ESL flagged, 16 dmg)

test2.esp (normal ESP, 12 dmg)

I added the item in the game, and it had 12 damage. So it appears that flagging an ESP as an ESL will allow it to function exactly as an ESP normally would by respecting the normal load order and conflict resolution, but with the added benefit of not taking up an entire plugin slot. Will do some further testing.

 

EDIT: Just did a couple more tests. I started a new game with a little over 350 plugins, with around 120 patches flagged as ESLs. The game loaded and ran fine, and I did a quick fly around Skyrim for 10 minutes with no crashes or visible issues. A good sign, I guess?

Second test was attempting this with Arthmoor's village mods. No dice. Most of them either CTD or have an ILS on start with the spinning loading wheel at the bottom right going on forever. The ones that do load, don't work properly in game (i.e. Dawnstar houses added by the mod having doors that aren't connected to anything). Everything works fine after removing the ESL flag, so it's pretty easy to revert back when testing plugins.

I'm guessing that flagging an ESP as an ESL only works if all of the records are overrides, with new records having issues... maybe?

 

EDIT 2: Also, just to be sure, the record limit for ESLs is 4096, correct?

Share this post


Link to post
Share on other sites

For implementing ESL support in xEdit properly, without having to dig through this whole thread and possible having to hunt for information elsewhere, can someone please summarize the following for me:

a) in as simple steps as possible, the exact way how to determine the correct load order

b) For determining the winning record if an ESL overrides a record from another module, is the priority determined by it's absolute load order, or by the position of it's renumbered form IDs?

That is, given the following load order:

full module -> [05]

esl module -> [FE 001]

full module -> [06]

if all 3 override a record from [00], will the esl module or the [06] full module win?

Share this post


Link to post
Share on other sites

For ESL files with the ESM flag on them, they will load after the official content for the game, and before any user files that have the ESM flag on them.

For ESL files with the standard ESP flag, I think more investigation needs to be done to be absolutely sure, but I recall these getting loaded after user made ESM files and before user made ESP files.

Regardless of which of the 2 blocks they end up in, they will be loaded into the game in alphabetical order.

As far as I understand it, your example with the ESL in between a pair of regular files should override in the order you have listed. You can't set an order like that though. The game will rearrange it automatically to fit the block scheme in the OP. In either case, 06 would win the conflict as far as I remember it.

Share this post


Link to post
Share on other sites

I went in and did some checking to be sure of how vanilla overrides behave in various files. This is the load order I used:

Quote

00  Skyrim.esm


01  Update.esm
02  Dawnguard.esm
03  Hearthfires.esm
04  Dragonborn.esm
05  ccBGSSSE010-PetDwarvenArmoredMudcrab.esl
06  ccBGSSSE014-SpellPack01.esl
07  ccEEJSSE001-Hstead.esm
08  ccBGSSSE035-PetNHound.esl
09  ESLMasterTestFileFirst.esl
0A  ESMMasterTestFileFirst.esm
0B  Unofficial Skyrim Special Edition Patch.esp  [Version 4.1.4]
12  ESLMasterTestFileLast.esl
13  ESMMasterTestFileLast.esm
14  ESPTestFileFirst.esp
5C  Alternate Start - Live Another Life.esp  [Version 4.0.7]
62  ESPTestFileLast.esp

There are gaps in the index numbers because I redacted the rest of the mods from the list for brevity.

When altering forn 00109112 (a book) the text I changed it to showed up as expected. 62 overrides everything else, 14 overrides all above it, 13 overrides, etc. The vanilla form is overridden exactly in the order these mods are positioned, and if they are rearranged differently, the override follows the new order.

So it seems as though however the mods show up in the mod manager's list (Wrye Bash Beta 3 in this instance) is how they'll be dealt with by the engine when they override vanilla forms. That also includes what happens when a Creation Club ESL file overrides a vanilla form. The order is still maintained by what the mod manager tells you.

Share this post


Link to post
Share on other sites
On 7/16/2018 at 4:35 AM, Arthmoor said:

I went in and did some checking to be sure of how vanilla overrides behave in various files.

Thanks a lot. Here is my current understanding of how load order works in all versions of the game engine that support .esl files (upper case ESM/ESL refers to header flag, lower case esm,esl,esp refers to file extension):

- Game.esm

- Official DLC in predefined

- CCC in predefined order

- All ESM flagged files (possibly also .esm and .esl files even if not ESM flagged, need clarification on that) in plugin.txt order (followed by all matching files that aren’t in plugin.txt in date/time order, as fallback)

- Possibly .esl files without ESM flag (need clarification if these form their own block or are mixed in above) in plugin.txt order (followed by all matching files that aren’t in plugin.txt in date/time order, as fallback)

- All remaining files (should only be .esp files, but I guess there is a remote possibility that it also includes .esm without ESM flag, need clarification on that) in plugin.txt order (followed by all matching files that aren’t in plugin.txt in date/time order, as fallback)

 

I would greatly appreciate it if you could help me clarify the following points:

.esm with out ESM flag - is it always considered to be ESM (and loaded accordingly)?

.esl without ESM flag - is it always considered to be ESM (and loaded accordingly)? Or is it considered to be like a .esp (without ESM flag) and mixed in with these? Or is there a group between ESM and not ESM where all .esl without ESM flag are loaded?

.esm or .esp with ESL flag - does that have any effect on load order (I don't think so). Is the file in both cases (.esm and .esp) assigned a "light" slot in regards to FormID mapping?

EDIT:

The list above only accounts for the order in which modules are shown to the user for selection (and in which the engine then tries to load the selected ones them).

The actual load process (AFTER confirming what modules you want to load) can result in further modification of the final load order, based on the following process:

Reduce the list to only the activated modules.

For each module in the list:

  Mark module as “being loaded”

  Get the list of masters, go through that list, and for each master:

    If marked as “load completed” ignore.

    If marked as “being loaded”, bomb out, modules are broken and contain a circular reference.

    Otherwise:

      Add (if not in the list) or move (if later in the list) the module in front of the currently “being loaded” module.

      Recursively repeat this process for the just added/moved module from the [Mark module as “being loaded”] line above before continuing

  Mark module as “load completed” and remove “being loaded” mark.

 

This process can potentially pull non ESM .esps in front of .esm/.esl files. As far as I understand it, this is the process that the game engine follows.

 

Share this post


Link to post
Share on other sites
1 hour ago, ElminsterAU said:

Thanks a lot. Here is my current understanding of how load order works in all versions of the game engine that support .esl files (upper case ESM/ESL refers to header flag, lower case esm,esl,esp refers to file extension):

- Game.esm

- Official DLC in predefined

- CCC in predefined order

- All ESM flagged files (possibly also .esm and .esl files even if not ESM flagged, need clarification on that) in plugin.txt order (followed by all matching files that aren’t in plugin.txt in date/time order, as fallback)

- Possibly .esl files without ESM flag (need clarification if these form their own block or are mixed in above) in plugin.txt order (followed by all matching files that aren’t in plugin.txt in date/time order, as fallback)

- All remaining files (should only be .esp files, but I guess there is a remote possibility that it also includes .esm without ESM flag, need clarification on that) in plugin.txt order (followed by all matching files that aren’t in plugin.txt in date/time order, as fallback)

 

I would greatly appreciate it if you could help me clarify the following points:

.esm with out ESM flag - is it always considered to be ESM (and loaded accordingly)?

.esl without ESM flag - is it always considered to be ESM (and loaded accordingly)? Or is it considered to be like a .esp (without ESM flag) and mixed in with these? Or is there a group between ESM and not ESM where all .esl without ESM flag are loaded?

.esm or .esp with ESL flag - does that have any effect on load order (I don't think so). Is the file in both cases (.esm and .esp) assigned a "light" slot in regards to FormID mapping?

From what I remember of my testing (about 7-8 months ago?):

  • .esm and .esl files get treated as if they have the master flag set even if they don't, i.e. it's set at runtime for those file extensions, so they go before .esp files.
  • I'm not sure if .esl files with and without the master flag set can be mixed, I don't recall testing this.
  • For libloadorder/LOOT I've appended plugins that aren't in plugins.txt in lexicographical order to avoid meaningless changes due to timestamps getting altered for whatever reason (which could mess up change detection). This is an implementation detail though, I consider the order to be undefined, so I guess it doesn't really matter.
  • I'm not sure about .esm's with the ESL flag set (I suspect they're treated as normal .esl's), but .esp's with the ESL flag set can be positioned amongst other .esp's and can have other .esp files as masters. I remember this wasn't initially the case but it got patched in at some point, I had to update my code. In terms of FormID mapping, the records in the .esp with the ESL flag that override others lose to overrides in later-loading .esp's, and new records appear at FE like normal for ESL files.

Share this post


Link to post
Share on other sites

As far as I can tell, .esl (new file extension) and ESL (new header flag) are much simpler than everyone seems to make them.

As far as I can tell there are only two relevant effects:

a) Files with the .esl extension are, like .esm files, considered to be ESM. This is purely based on extension, and it has exactly the same effect on load order as .esm as extension.
b) Files (no matter the extension) with the ESL flag get assigned the next “light” slot instead of the next “full” slot when being loaded. (Need to clarify if .esl files without ESL flag are treated as ESL anyway).

That’s it. That’s the only difference.

And the only effect of having a light slot vs. a full slot is how FormIDs are mapped between global and local (specific to a file) FormIDs.

FormIDs belonging to a “full” module are mapped as <FullSlot: 1 byte><ObjectID: 3 byte>
FormIDS belonging to a “light” module are mapped as <FE: 1 byte><LightSlot: 1.5 byte><ObjectID:1.5 byte>

Other than that, they behave exactly the same as any other module.

And you have exactly the same level of control over when .esl files load as you have over .esm files. Both are considered, based on extension alone, ignoring the header flag, to be ESMs, and they will load in plugin.txt order before any non ESMs. Based on plugin.txt order, they can load before, after, among, any other ESM (which means .esm, .esl, or ESM flagged files).

 

 

Share this post


Link to post
Share on other sites
9 hours ago, ElminsterAU said:

As far as I can tell, .esl (new file extension) and ESL (new header flag) are much simpler than everyone seems to make them.

As far as I can tell there are only two relevant effects:

a) Files with the .esl extension are, like .esm files, considered to be ESM. This is purely based on extension, and it has exactly the same effect on load order as .esm as extension.
b) Files (no matter the extension) with the ESL flag get assigned the next “light” slot instead of the next “full” slot when being loaded. (Need to clarify if .esl files without ESL flag are treated as ESL anyway).

Also c) you can activate 4096 of them for the cost of only 1 normal plugin slot.

libloadorder only cares if a plugin is a light master or not when it:

  1. checks how many light masters you have active
  2. decides where to insert a plugin (if it's a light master that doesn't have the .esp extension, it goes before ESPs)

So I'd agree that it's simple, but it wasn't simple to get to that understanding, and the explanations are probably reflecting how our knowledge progressed and regressed as we tested things and as the implementation changed.

Share this post


Link to post
Share on other sites

that's not really a c), that's the consequence of b).

there are 254 available "full" slots ( 00 to FD ) (FE is for light modules, FF for the save game)

and there are 4096 available "light" slots (FE 000 to FE FFF)

modules are being loaded in the order that I described above and are assigned the next available full or light slot based on the ESL flag in the header. If there is available slot of the right type available anymore when a module is being loaded, you get an error.

and your point 2 is also already covered in my a). .esm and .esl files are always considered to have the ESM flag set, even if they don't. All files with (implicit or explicit) ESM flag are initially placed in the load order before any files without ESM flag.

That's it.

You can pretty much reduce the required information about .esl/ESL to:

- .esl files are implicitly ESM flagged, they sort as such

- ESL flagged files get assigned a light slot instead of a full slot, as such they can only contain ObjectIDs in the 000 to FFF range. There are 254 full and 4096 light slots available. FormIDs for light slots are mapped as <FE: 1 byte><LightSlot: 1.5 byte><ObjectID:1.5 byte>

The latest version of xEdit that you can find at https://github.com/TES5Edit/TES5Edit/releases should fully support ESLs now. (The load order code still needs some minor work.)

Share this post


Link to post
Share on other sites

Just a heads up - information about this subject has changed quite a bit as new understanding was obtained. I've rewritten the OP to reflect this as I understand it all now.

Share this post


Link to post
Share on other sites
On 10/6/2017 at 11:16 AM, Arthmoor said:

A .esp file can function as either type of file depending on which flags are set in its file header. If the ESM flag is set, then the .esp is treated as a master file. If the ESL flag is set, it will be treated as a light plugin. If BOTH flags are set, then it is treated as a light master.

Why does the ESL flag win over the ESM flag?  Is that Bethesda's decision?

Share this post


Link to post
Share on other sites
1 hour ago, Leonardo said:

Why does the ESL flag win over the ESM flag?  Is that Bethesda's decision?

It doesn't, the flags are orthogonal. The ESM flag controls if it's a master, the ESL flag controls if it's light.

Share this post


Link to post
Share on other sites

Sorry if this is dumb, but I haven’t found an answer anywhere else: I have an .esp file with 94k records on a released mod. I’ve been told there can be problems when the record count reaches the upper 100k’s, and the plan is to add more content.

Would converting to .esm significantly help with record count capacity, and what types of problems could emerge as a result of changing the extension from .esp to .esm? (assuming anyone with the .esm would be warned to use it on a save game that hasn't seen the .esp version before).

I bring it up because Far Harbor and Nuka World are both around 300k records as .esm’s, with Dawnguard at 100k and Dragonborn at 250k, but I’m struggling to find examples of .esp files over 100k records.

Share this post


Link to post
Share on other sites

I don't think there's a specific issue with the number of records, but with the types of records. Larger .esp files will tend toward problems when being edited in the CK but I've never heard of such a limit for raw record count. It would take a hell of a lot of stuff to reach counts that high anyway. At least as large as Dawnguard.

Share this post


Link to post
Share on other sites

Thanks for your reply.  "but with types of records..." I'm confused on that part. Which types of records are worse? Most of the records for the file in question are static objects used in custom interior cells.  "problems when being edited in the CK": the noticeable difference in CK for this esp file is that it takes longer to save, around 10 seconds.

Share this post


Link to post
Share on other sites

I've read the whole thread after @alt3rn1ty linked it in the CBBE comments, where a discussion on this topic was going on. CBBE 1.5 includes an .esl to correct some vanilla clothing paths to apply the new edge lighting effect. My issue with Wrye Bash and .esl's is how they affect the save files, mainly. I'm just trying to understand what is going on and if there's a problem or if can just ignore it. I'm using Sharlikran's latest build at the moment, was using the latest from the second post in WB thread before yesterday.

I have two .esl files:

CBBE.esl, which Bash indicates is a master flagged as ESL. It goes in slot 09 after the other masters and can't be moved down (it can be moved up).

CosmeticVampireOverhaul - Cutting Room Floor Patch.esp, which is also flagged ESL. It can be moved anywhere, currently at 42.

My save files end up orange. The masters list shows cbbe.esl at position 09 but its MI is 4C. CVO-CRFP.esp, on the other hand, is shown all the way at the bottom at 4D, which is its MI but its current LO is still 42. I don't understand the technicalities, I just want my saves purple again. I'm wondering what's the best course: avoid .esl's altogether or ignore the way WB currently shows a save's masters.

Share this post


Link to post
Share on other sites

I turned some weapon mods to ESPFE (ESP with the ESL flag set, the file extension is still .esp). Apart from that the ESP and the ESPFE version are completely the same. No form IDs were changed (and yes, all of them are between xx000800 and xx000FFF). What happens is that both files working in game as they should except the dual sheath does not work with the ESPFE file. Again: the only difference of both mods is the ESL flag set in the file header.
 
So, what's going on? The dual sheath mod is essentially dependent on the weapon mod. I assumed the ESPFE file behaves like a normal ESP. However it seems the form IDs change when the ESPFE mod is loaded. This would mean any mods that are dependent on the mod are not going to work when the mod is an ESPFE. When this the case actually than it is a huge drawback. Especially weapon mods were good candiates to turn them to ESPFE but without dual sheath it does not make much sense.

Any ideas about this? I case I am right, it is worth to update the sticky I think.

Share this post


Link to post
Share on other sites

Hilli1, I would assume this is because DSR is not an ESL and the "patch" is being read before the weapons are being integrated. Typical load order issue there. You would need to have the DSR patch be an ESL and load it after the weapon mod. At least if my understanding is correct on how the ESL work.

 

In my case though, I'm trying to create a super large mod load (over 700 plugins after patches). And I am really just finding out about the whole ESL structure. I was wondering, are there any types of mods that should NOT be converted into an ESL? If I'm wanting to make these plugins work together without merging them, I'm looking to figure out which mods I should convert and which I should leave without the ESL flag. This load out I would normally just start merging patches and changing masters as necessary to pare it down. But if I can save time and stability by converting most of these to ESL, that would probably be ideal. I'm aware I'll still need to do the heavy patch work for conflict resolution and such, but cutting out the merging process would probably be better?

Share this post


Link to post
Share on other sites

Grab one of the test builds of Wrye Bash off of their Discord channel. Those will indicate mods that are safe to flag as ESL in green.

As far as what not to convert, my own preference is to only do so for patches between mods and for primary mods that are very small or only alter vanilla records.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Support us on Patreon!

×