Jump to content
Arthmoor

Plugin Files And You (ESM/ESL/ESP)

Recommended Posts

Beginning with Patch 1.5.3 of Skyrim Special Edition, the game supports a new type of mod file known as a "light plugin" with the .esl extension. I've spent some time going over these files now that Wrye Bash is able to display the save headers and shown how they are listed based on how they are flagged in your load order. It's somewhat of a convoluted mess, but there is a certain order to it as well. Things are divided up into 6 distinct blocks of files according to the save headers. This post will serve as a complete breakdown of how all types of plugin files will load into the game and be listed in your save headers.

Block 1: The original 5 ESM files. The load order of these is hardcoded and cannot be changed. The game will always load them. If any are missing, the game will crash on startup.

Block 2: Standard ESM files produced by modders. These begin loading immediately following Dragonborn.esm. The load order is as listed in your mod manager. This block also covers ESP files which are flagged to act as ESM files, such as the USSEP. They must be listed as enabled before the game will load them.

Block 3: Creation Club content ESL files flagged as masters. This block is loaded after the final standard ESM file. The order of these files is hardcoded to load alphabetically. Even if these files are marked as disabled, the game will still load them. You must completely delete one in order to remove it from your game.

Block 4: ESL files created by modders which are flagged as masters. These will follow the load order as listed in your mod manager. These must be enabled for the game to load them.

Block 5: Standard ESP files. What most mods will actually be. These are the normal types of files generated by the CK when modding. They will load in the order specified by your mod manager and must be marked as enabled for the game to load them.

Block 6: ESL files converted by modders in the CK. They will be loaded according to what's set in your mod manager.

When an ESL of any type is loaded at runtime, the game converts all of the form IDs inside to use the FE index slot. This very likely means that this slot, once useable by normal mods, is now considered off limits. Results of trying to load an ESP in this slot while also having ESL files installed has not been tested and is probably not going to work.

Attempts to make ESL files that use standard ESP files as their masters have so far all resulted in the ESP files being bumped to load into the Block 4 area instead of Block 5 where they belong. This is highly undesirable since it will cause problems with the ESP files being out of their expected positions in a functioning load order. This can lead to anything from edits being overridden in unexpected ways to the game crashing. Using the console to check the runtime form IDs of the ESPs that get bumped is confirmed so it would be a very bad idea to ignore this.

ESL files also have other important restrictions to consider, which are detailed here: https://www.creationkit.com/fallout4/index.php?title=Data_File

Share this post


Link to post
Share on other sites

I've tested having a normal ESP in the FE slot with ESL plugins present, and if the normal plugin and an ESL have a record with the same Object ID, the ESL record is used. The other record is replaced, it doesn't get remapped to a different FormID. Otherwise, non-conflicting records share the FE index.

Also, if you have over 4096 ESLs, their records overflow into FF.

Share this post


Link to post
Share on other sites

User-created ESLs flagged with ESM can load above or below user-created ESLs not flagged with ESM; Plugins.txt dictates this.

ESLs can also load before USSEP.  The in-game mod load order menu will default ESLs to right below USSEP and will not allow ESLs to be moved below normal ESPs.  It will allow them to be moved above USSEP.   USSEP wins conflicts with ESLs above it, and loses conflicts with ESLs below it.  (USSEP is the only ESM-flagged ESP mod I have, but I assume this would be true for other such mods; not sure about user-created ESMs as I do not run with any currently.)

Is there any reason at all user-created ESLs be flagged ESM?  This could be reserved to distinguish official CC ESLs, as they will always be flagged ESM.

 

Share this post


Link to post
Share on other sites

To make sure I understand the load-order implications of ESL files correctly: What happens if there is a conflict between an ESL flagged as a master and a normal esp?

Will the game detect the conflict and leave out the conflicting record when merging all the ESL files into FE (meaning the ESP wins) or will it write the ESPs version of the conflicting record into FE or will the ESL file win the conflict, despite being in a higher block according to the save file headers?

Share this post


Link to post
Share on other sites
4 hours ago, Daidalos said:

To make sure I understand the load-order implications of ESL files correctly: What happens if there is a conflict between an ESL flagged as a master and a normal esp?

Will the game detect the conflict and leave out the conflicting record when merging all the ESL files into FE (meaning the ESP wins) or will it write the ESPs version of the conflicting record into FE or will the ESL file win the conflict, despite being in a higher block according to the save file headers?

The ESP's version gets used as the FE record.

Share this post


Link to post
Share on other sites
6 hours ago, Nukenin said:

User-created ESLs flagged with ESM can load above or below user-created ESLs not flagged with ESM; Plugins.txt dictates this.

ESLs can also load before USSEP.  The in-game mod load order menu will default ESLs to right below USSEP and will not allow ESLs to be moved below normal ESPs.  It will allow them to be moved above USSEP.   USSEP wins conflicts with ESLs above it, and loses conflicts with ESLs below it.  (USSEP is the only ESM-flagged ESP mod I have, but I assume this would be true for other such mods; not sure about user-created ESMs as I do not run with any currently.)

Is there any reason at all user-created ESLs be flagged ESM?  This could be reserved to distinguish official CC ESLs, as they will always be flagged ESM.

 

The problem here is that it doesn't matter where you tell it to put them, the results will be as listed in the OP. The save headers will come out with them listed in block order according to the plugin type. I've already tried numerous combinations like you specify and they always came out listed in the same groupings and things were not being overridden in the manner you are describing when examining the form IDs in the console for objects in the world. So it very much looks like a case where ESLs mapping to FE means that where they show up in the load order list is absolutely meaningless.

This means they won't be suitable for use as anything more than simple stuff, such as mods that alter vanilla records exclusively, or mods that only add their own new records and don't conflict with where they get placed into the game. Trying to account for ESLs in a bashed patch would be meaningless too and likely cause problems.

Share this post


Link to post
Share on other sites
38 minutes ago, Arthmoor said:

This means they won't be suitable for use as anything more than simple stuff, such as mods that alter vanilla records exclusively, or mods that only add their own new records and don't conflict with where they get placed into the game.

So both ours map marker mods can be or should be created as ESL, at least your Dawnguard Map Markers mod.  And it is understable why you would convert it to ESL due to the loadorder structure Bethesda introduced in the 1.5.3 patch.

Share this post


Link to post
Share on other sites

Yes, USSEP has been confirmed by form IDs in-game to load right where it's expected in the 05 slot. All of the ESL content shows up as FE, so the ESLs will override everything above that when there's a conflict.

Share this post


Link to post
Share on other sites
Posted (edited)

I was able to get USSEP to override ESLs placed above it in load order.   I was testing with two dead-simple ESLs that modified Dragonbane—one zeroed its enchantment cost (and enchantment, to remove the bar), while the other just changed its name.

Placing both above USSEP resulted in USSEP's Dragonbane (original name, charges/enchantment) winning the conflict.

I'm not doing any deep investigative work, just anecdotal goofin' 'round.  At varying points one or the other of the aforementioned ESLs was also flagged with ESM, but I never really did a thorough breakdown of what was happening when as I was already borrowing time I needed to spend on the paying work. :)

My interest is purely load order and not where things end up in data structures*.  Things win or lose based on when they load, not where they end up.  Is that too simplistic?  (And if it's wrong, let me have it!)  [EDIT: Well, of course I'm intrigued by data structures; I just have the (possibly mis-?)perception that the where is just a consequence here, and not the arbiter.]

I apologize for coming at this sideways; I'm selfish, I want to be able to use SSEEdit and LOOT without freakouts or wacky error messages just because I might want to patch one of the CC ESLs with an ESP.  I've avoided modding Skyrim for six years (as a mod creator, that is) but every time I stray into the periphery of these technical discussions it threatens to stir my interest.

Also, I suspect that the CC ESLs loading alphabetically might just be coincidental fluke because they are hardcoded in the EXE alphabetically.  (Something governs their automagic loading, and it's not just having a filename beginning with "cc" nor is it having the ESM flag set.)  I guess we'll have a better insight when the next round of CC content hits.

 

Edited by Nukenin
conflict resolution

Share this post


Link to post
Share on other sites

Nukenin, my interest is stirred as well.  However, I have no idea why Bethesda would create CreationClub and the .esl data structure.  Isn't the modding community working on the current framework?  I just watched a video on Creation Club - new weapons, armor, blah, blah, blah.  If ESL's are easier for new modder's to create new content, that makes sense.  Of course, I find the current modding paradigm working just fine to add new content to Skyrim and Skyrim SE.

Is Creation Club what jquery is to web development?  Curious.  Edit: I just watched GamerPoets  "I can't sleep, so I'll talk about Creation Club and paid mods...sort of." video on YouTube, and it seems that Creation Club and the ESL paradigm is not the same as jquery for web development. 

Jquery doesn't cost anything, and enhances web development.  Creation Club is more of a "stable" mods that you can purchase using credits and the mod is compatible with current saves.   Gopher's youtube video is hilarious but he did give some credit to this new format.  Creation Club is similar to jquery where you can select mods you want, and they will work with current saves.  Instead of searching Nexus for mods, the player just selects through the in-game menu for what they want and install it.   Thoughts?

I'm still wondering why change the paradigm because I think there are some really talented modders - hello Arthmoor! - and I'm not sold on adding another data structure (ESL) to a working mod paradigm.

Strat.

Share this post


Link to post
Share on other sites

@Arthmoor: The wiki page contradicts your posted info on where CC ESLs load: the wiki says they load before modder ESMs, you say after. I haven't checked which is right myself, but the wiki page makes CC ESLs consistent with other Bethesda-supplied plugins.

*Edit:* I pressed enter trying to @mention and it submitted my post... Also, does anyone know how the variable limit on the number of active ESLs arises? The wiki page says it varies with the number of records each ESL contains, but I don't see why that matters, and no further details are given.

The wiki page also contradicts the OP by saying there's no difference between ESLs with and without the master flag, they all load before ESP files.

Share this post


Link to post
Share on other sites

Yeah, sorry, I was the one who edited the wiki to say that before Bash was reading the save headers and cleared things up. I guess I should go fix that.

Share this post


Link to post
Share on other sites
23 hours ago, Arthmoor said:

Yeah, sorry, I was the one who edited the wiki to say that before Bash was reading the save headers and cleared things up. I guess I should go fix that.

The save headers are misleading though, they place all ESLs after all other plugins - I was trying to replicate the OP's statements using the save headers to see what was happening, and the save header contains:

Skyrim.esm
Update.esm
Dawnguard.esm
HearthFires.esm
Dragonborn.esm
Blank.esm
Blank.esp
ccqdrsse001-survivalmode.esl
Test Master.esl
test.esl

Where Blank.esm/Blank.esp are plugins with no masters and the appropriate master flag values, Test Master.esl is a copy of  ccqdrsse001-survivalmode.esl with the localisation flag removed, and test.esl is a copy of ccqdrsse001-survivalmode.esl with the localisation and master flags removed.

The above doesn't make any sense except as the load order after all ESLs have been mapped to FE, so doesn't describe the pre-mapping load order as I'm assuming the OP is trying to. It also doesn't match the results of the testing I did when ESLs initially hit FO4, but I haven't retested them against Skyrim SE 1.5.3.

@Nukenin's tests reported in the TES5Edit thread check using record overriding as mine did, and his findings are that:

  • ESLs load in their plugins.txt order amongst themselves
  • ESLs load before plugins without the master flag set
  • Non-CC ESLs can load before or after (non-harcoded) plugins with the master flag set, depending on plugins.txt order
  • CC ESLs always load before other ESLs and (non-hardcoded) plugins with the master flag set

However, it's unclear if the ESL plugins he made had the master flag or not. Please correct me if I've misunderstood anything, Nukenin.

Now, I previously found but have not re-verified that:

  • CC ESLs load immediately after the other hardcoded plugins, as it's both more sensible and easier to implement (and why would the devs make their own lives harder for no obvious benefit?).
  • Whether or not an ESL had the master flag set had no effect on where it loaded relative to non-master-flagged plugins.

It seems to me that for the purposes of tools interacting with load order, ESLs are just like master-flagged plugins, to be treated no differently - they always go before non-master-flagged plugins, and can intermingle with master-flagged plugins (excluding hardcoded plugins). So using the blocks defined in the OP, I think it goes:

  1. Block 1
  2. Block 3
  3. Blocks 2, 4 and 6 possibly all intermingled
  4. Block 5

It would be good if someone could independently verify or disprove this.

Share this post


Link to post
Share on other sites

I've been traveling so apologies for getting back a bit late on this.  I know at some point I fiddled with setting ESM on the ESLs I was testing with, but I don't recall it making a distinction in the load order.  But I don't trust my memory enough to say that with certainty. :)

Share this post


Link to post
Share on other sites
23 hours ago, Nukenin said:

I've been traveling so apologies for getting back a bit late on this.  I know at some point I fiddled with setting ESM on the ESLs I was testing with, but I don't recall it making a distinction in the load order.  But I don't trust my memory enough to say that with certainty. :)

That's OK, I did some more testing and answered most of my questions.

Share this post


Link to post
Share on other sites

From a post by CDCooley https://forums.nexusmods.com/index.php?/topic/6043958-unofficial-skyrim-creation-club-content-patches/page-15#entry54334023

Charlie breaks it down to 4 groups ..

Quote

There's absolutely nothing magical happening here, but people have gotten too comfortable with the idea that those first two digits of a formId you see in the game's console represent load order. They used to but now they don't thanks to ESL files. It's time to adapt folks. I'm seeing various people testing how things load by looking that the formID numbers in-game. 

Load order is controlled by the game's hard-coded list of official content and then things listed in the plugins.txt file. The numbers assigned as the first two digits of formID values in-game have nothing to do with load order when ESL files are involved. 

From my testing there appear to be only four groups of files that the game considers when loading files. 

First are the official files that are hard-coded into the game itself. Those used to be just Skyrim.esm, Update.esm, Dawnguard.esm, HearthFires.esm, and Dragonborn.esm but now include all of the new Creation Club ESL files too. That means the survival ESL will always load before any traditional mods including USSEP. If they exist in your Data folder they will be loaded and any entries in the plugins.txt file for them are ignored. 

Second are files listed in plugins.txt that are internally tagged as masters and marked as active. It doesn't matter what file extension they use (ESM, ESP, or ESL). They get loaded in the order they appear in the plugins.txt file (but only their order in relation to each other matters). 

Third are any other mods as listed in plugins.txt that are marked as active. The are the regular mods that we used to call ESPs but can technically include files with the extensions ESM and ESL too. 

And finally, the fourth group consists of only one file. It's the save file (ESS) being loaded (or a blank file if you're starting a new game).

Hopefully various mod managers (and LOOT) can be updated soon before too much chaos results from people thinking there's something more complicated and confusing happening. 

 

Edit : My bold in yellow

Share this post


Link to post
Share on other sites

This is starting to get weird. I think what needs to happen at this point is someone needs to come up with a small series of tests to ferret out just how this all works. The stuff I did was fairly basic. Official CC files, a master flagged ESL, and one regular ESL. It seems clear we need a few more cases to try out and gather results from. Like, say, a master flagged ESL using a CC ESL as a parent, a standard ESL using CC files as parents, normal ESLs using each other as parents, and so forth.

Either that or someone needs to go bug Bethesda for a proper official explanation of how all this works because despite various people testing things, one thing still seems clear to me - load order index numbers as shown in Wrye Bash are where the standard ESM and ESP files get mapped to in-game because clicking on their stuff in the console demonstrates that much.

Share this post


Link to post
Share on other sites

A test matrix to track what has been tested and what needs to be tested would be an excellent thing. If it needs to be done I can create one in Google Docs and share it out.

Also regarding the more technical specifics of this, back in the DOOM/Quake days we used to just email John Carmack (or Romero or John Cash) to get technical details when we couldn't figure something out.

Is there anyone at Bethesda that could help or at least direct us to someone that could answer the question?

Share this post


Link to post
Share on other sites
1 hour ago, Beermotor said:

Is there anyone at Bethesda that could help or at least direct us to someone that could answer the question?

I could be wrong. If something similiar happen on the old BSF then SmkViper (not sure if he is still he working for Bethesda or not) is the one the community could try to contact, because he was probably the only one who was fairly active in the forum.

Share this post


Link to post
Share on other sites

I ran a few tests to check stuff that Arthmoor, CDCooley and I disagree on:

Quote

Edited an item's name so that it had the following suffixes:
    Master ESL
    Non-Master ESL
    Master ESM
    Non-Master ESP

Then spawned the coat using player.additem and checked its name. A clean save was used each time.

The items were:
    Fallout 4: 001BBCB5 (Armor_Railroad01)
    Skyrim SE: 000CC842 (AncientNordAmulet)

Order 1:

    *non_master.esl
    *master.esl
    *master.esm
    non_master.esp

    The coat was called Master ESM, so ESLs respect plugins.txt relative to ESMs, ignoring master flag.

Order 2:

    *non_master.esl
    *master.esl
    master.esm
    non_master.esp

    The coat was Master ESL, so ESLs respect plugins.txt relative to each other, ignoring master flag.

Order 3:

    master.esm
    *non_master.esp
    *non_master.esl
    *master.esl

    The coat was Non-Master ESP, so ESPs always load after ESLs.

Order 4:

    As Order 3, but I edited the non_master.esl to remove its 0x200 flag.

    The coat was Non-Master ESP, so the 0x200 flag does not determine whether or not the game treats a plugin as a light master.

 

Share this post


Link to post
Share on other sites
11 minutes ago, WrinklyNinja said:

I ran a few tests to check stuff that Arthmoor, CDCooley and I disagree on:

Edited an item's name so that it had the following suffixes:
    Master ESL
    Non-Master ESL
    Master ESM
    Non-Master ESP

Then spawned the coat using player.additem and checked its name. A clean save was used each time.

The items were:
    Fallout 4: 001BBCB5 (Armor_Railroad01)
    Skyrim SE: 000CC842 (AncientNordAmulet)

Order 1:

    *non_master.esl
    *master.esl
    *master.esm
    non_master.esp

    The coat was called Master ESM, so ESLs respect plugins.txt relative to ESMs, ignoring master flag.

Order 2:

    *non_master.esl
    *master.esl
    master.esm
    non_master.esp

    The coat was Master ESL, so ESLs respect plugins.txt relative to each other, ignoring master flag.

Order 3:

    master.esm
    *non_master.esp
    *non_master.esl
    *master.esl

    The coat was Non-Master ESP, so ESPs always load after ESLs.

Order 4:

    As Order 3, but I edited the non_master.esl to remove its 0x200 flag.

    The coat was Non-Master ESP, so the 0x200 flag does not determine whether or not the game treats a plugin as a light master.

Good job! Seems like good news - I guess the Orders are the plugin.txt entries ?

1. games behave the same

2.  The rest basically means we have to force load the esls (based solely on extension)  with the esms and let plugins.txt decide amongst them ?

Share this post


Link to post
Share on other sites
14 minutes ago, Utumno said:

Good job! Seems like good news - I guess the Orders are the plugin.txt entries ?

1. games behave the same

2.  The rest basically means we have to force load the esls (based solely on extension)  with the esms and let plugins.txt decide amongst them ?

Yes, yes and yes.

My outstanding questions are:

  • Can a normal plugin and a light master share the FE slot? (They could in Fallout 4 when I tested in early September, but that might have changed.)
  • What's the precise relationship between number of records in ESLs and how many can be active?

I can test the first question myself, but the latter would be painful to answer with black-box testing, and could really do with some clarification from a dev or someone who can poke around the runtime innards to figure it out.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Support us on Patreon!

×