Error writing metadata

Troubleshooting and Bugs
1

System notes:
Negative Lab Pro 3.0.2
“Negative Lab - Sony ILCE-6700 v2.3.dcp” (custom profile provided by @nate — see here)
Lightroom Classic 13.1
MacOS Sonoma 14.2.1

General description of the problem: With a fresh Lightroom catalog and newly imported B&W scans done using my Sony a6700, I’m getting an error writing metadata immediately after opening NLP, which persists after conversion, and which can cause other problems (see below). This happens when one or multiple negative images are selected.

Steps to reproduce:

  1. Select one or more negatives in Lightroom.
  2. Hit Ctrl-N to open NLP.
  3. Before converting or changing anything at all, I immediately get the exclamation mark icon in Lightroom, with a tooltip that says “Error writing metadata”.
  4. If I cancel out of NLP, without making any changes or converting the images, I can click on the exclamation mark and get this pop-up:
    Screenshot 2024-01-01 at 1.52.55 AM
  5. If I click on “Import Settings from Disk”, the error seems to resolve.
  6. Instead of cancelling out of NLP, if I go ahead and convert the negative, then the exclamation mark remains after conversion.
  7. If I click on the exclamation mark, I get the error pop-up again. If I then click on “Import Settings from Disk”, then the converted positive image reverts to a negative image.
  8. If I then reopen NLP, before changing any settings, the image redisplays as positive.
  9. Hit “Apply”.
  10. At this point, one of two things can happen. Either everything seems to return to normal, or the exclamation mark and the metadata writing error will return. (The error seems more likely to return in batch conversions, but I’m not sure about this, maybe more files means higher probability of there being an issue.)
  11. If the error returns and I try to import the metadata from the disk (from the error popup) or “Read metadata from file” in LR’s menus, sometimes the error will resolve. However, sometimes things will deteriorate. Sometimes, I will sometimes get a different error writing metadata popup, listing failed file writes for various files. In the case of one catalog (470 images), the error caused irrecoverable catalog corruption. I had to start a new catalog and re-add the DNGs.
  12. If I try to un-convert the images with errors, LR will sometimes get “confused” and display un-converted images as positive images. At this point, I can re-convert and reset the images (using NLP’s reset option), but this will also cause the exclamation mark / “error writing metadata”.

Other notes:

  1. With multiple images with exclamation marks, I cannot seem to read the metadata from multiple files at once in order to resolve the error. I have to deal with each file individually, which is pretty tedious.
  2. I installed NLP using the “run the installer from the Applications folder” method (recommended for more recent versions of MacOS).

Things I have tried: restarting LR, restarting the computer, reinstalling NLP, reinstalling LR, manually removing NLP and reinstalling it, starting from a fresh catalog.

Can anyone help with this problem?

2

I’ve never had such a case if I remember correctly, but if I had, I’d check access rights of those files (easy to do with Terminal.app, if you’re comfortable with a CLI) or try the following: In Finder, duplicate the folder containing the problematic files, import the new folder with Lightroom and retry NLP.

Duplicating the folder should set the correct access rights, should they be the problem. If the issue persists, I’d try loading the folder from a backup and test with that folder again. Chose the oldest backup of the folder. If the new or restored folder solves the issue, you can delete the old folder after a few days (wait a few days, just in case)…

Moreover, you could run disk utility and check your volumes.

Update, attn. @nate

Did a few tests with a newly set structure of folders and images and got the “!” badge too.

Launching NLP v3.0.2 produces the badge on a seemingly random number of images. Repeating a few times makes the badge appear on different or the same images and I cannot make out a pattern which files get the badge. Looking at access rights in Terminal with ls -n, I got this:

-rwxrwxr-x@ 1 501  20  38914290 25 Feb  2020 B01_20170919_0002.cr2
-rw-r--r--  1 501  0       8483  2 Jan 00:05 B01_20170919_0002.xmp
-rwxrwxr-x@ 1 501  20  41019320 25 Feb  2020 B01_20170919_0003.cr2
-rw-r--r--  1 501  0       8483  2 Jan 00:05 B01_20170919_0003.xmp

We can see that the image and sidecar files belong to user ID 501, but group and world membership or access rights differ: 775 for images, 644 for sidecars with their respective group memberships.
While this does not seem to bother Lightroom at all, I get the feeling that NLP is accessing the files with other credentials (user or group membership) - or possibly with a timing that disagrees with Lightroom or something else. The situation is independent of whether I keep the images in the shared user folder or my own (I’m user ID 501)

Canceling NLP does not remove the badge, but reading from or writing to the file(s) removes it. Due to the absence of the badge in former tests, I suspect that there is something in NLP that needs to be fixed in order to not cause the “!”.

Tested and found with NLP 3.0.2 on macOS 12.7.2 on 5K iMac 2019.

3

I ran First Aid in Disk Utility. Everything checked out. File permissions are 644 on all files, before and after opening NLP and conversions.

I also ran a new test. I moved 36 ARW images to an external disk and started a new catalog, also on the external disk.

I imported the first 12 ARWs as DNGs and hit Ctrl-N to open NLP. Once again, before changing any settings, all 12 images got an exclamation mark, i.e., “error writing metadata”. I left these files unconverted. Remember this for later on…

I then imported another 12 ARW files, but left them as ARW files. After opening NLP, no exclamation marks. After converting them, still no exclamation marks. The problem does not seem to affect ARW files (because they don’t contain NLP metadata)!

File permissions were 644 on all files, before and after opening NLP and conversions. I don’t have sidecar files, all edits are in the catalog.

→ I then turned on “Automatically write changes into XMP” in the catalog settings. This of course created XMP files for the ARW files.

I then selected the original 12 DNG files, all of which had exclamation marks. I opened NLP and converted them. The exclamation marks disappeared after conversion.

I could then edit the files in NLP without any issues. File permissions remain 644 throughout.

I then imported a third set of 12 ARW files and converted them using NLP. No exclamation marks. I then tried to convert these to DNGs in LR, but the convert to DNG pop-up locked up, i.e., I couldn’t change any settings or click on OK or Cancel. I had to ESC out. This may be unrelated. Restarting LR, I then converted these files to DNGs Still no exclamation marks. I then un-converted these files in NLP and initially got exclamation marks that disappeared on their own after a second or two (without intervention on my part). I then re-converted these DNGs in NLP and every image got an exclamation mark, which again disappeared after a few seconds. I then un-checked “Automatically write changes into XMP” in the catalog settings and the exclamation marks returned!

Summary: In my case, the problem seems to occurs with DNGs, but not with ARWs, and setting “Automatically write changes into XMP” seems, superficially, to resolve the problem. Or more precisely, the problem is still occurring (the exclamation marks appear initially), but it looks like the forced XMP writing seems to clear it up.

Other notes that may or may not be relevant:

  • I don’t know whether using the “install NLP from the Applications folder” method changes the way it handles permissions when used. More generally, there may still be a permissions problem somewhere.

  • These are all B&W images.

  • My files have a certain amount of not-necessarily-usual metadata baked in because I’m scanning 2-1/4 square negatives and using a 1:1 aspect ratio in camera in order to help with framing and to save a step in LR. They get imported as 3:2 images pre-cropped to 1:1.

4

Here’s a hypothesis, maybe someone can substantiate it or invalidate it. If this problem (“error writing metadata”) is not a disk error or a permissions problem, then it occurs to me that NLP is somehow generating a “mismatch” between the catalog metadata and the DNG metadata (or maybe between the catalog metadata and the XMP files in @Digitizer’s case). This “mismatch” would then be getting interpreted by LR as an “error writing metadata”.

If this (or something like this) is the case, then of course enabling “Automatically write changes into XMP” in the catalog settings “corrects” the mismatch because this option forces LR to write the XMP information to the DNGs (or XMPs). The “mismatch” would thereby get “corrected” (as it were, because the mismatch shouldn’t be happening in the first place).

@Digitizer: Do you have “Automatically write changes into XMP” enabled or disabled in your catalog settings? And is there a difference between your test catalog (which maybe didn’t have that option enabled) and other catalogs in which the error doesn’t occur?

5

I set Lightroom to NOT sync XMP automatically and both test folders were tested with the same Lightroom catalog.

But I’ve discovered something:
When I change the group of the surrounding folder and its content, NLP will work without the “!” badge. When I set group to “staff” (group ID 20) instead of “wheel” (group ID 0), the issue dissolved.

You can check and change rights using the “Information” window that you can get trough the context menu. ctrl-click on a folder and select “Information”, rights are listed in the bottom drawer of the popup.

6

@Digitizer: All my files are 501:20 (my userid, group “staff”) by default, with 644 (-rw-r–r–) permissions, also by default. So we could be seeing two different issues here, or maybe two manifestations of the same issue (with different file system parameters). Could you perhaps humour me and create a new catalog with a few images, starting with “Automatically write changes into XMP” disabled, then do whatever creates the error for you, and then finally enable writing to XMP, and see if that clears the exclamation marks?

7

Okay, this is what I did, all of it in Lightroom

  • backed up the Lr Catalog
  • created a new folder as a sibling to the test folder
  • exported CR2 files from the test folder to the sibling
  • tried NLP with the copies of the CR2 files from the sibling folder → “!”
  • tried NLP with the same CR2 files from the test folder → “!”
  • changed group and rights, which did not seem to help
  • had to restart LR from a database backup in order to set things straight.

Please check if your system does the same thing.

Having to restore the catalog makes me nervous. Inconsistencies in the DB? Hard to say/believe but can possibly not be excluded. But if there were inconsistencies, the “!” would appear on other occasions and always on the same files. Have to explore some more. Aaargh!

8

@Digitizer: Yes, that’s exactly what happens in my case too (except, as mentioned, my file permissions are all 644 and all my files get exclamation marks after NLP launch). Would you be open to trying the following on your machine?

  1. Backup your usual catalog, of course.
  2. Copy (cp) a few of your photos and their associated XMP files to a disposable directory, like ~/Downloads/test/photos/ or something. Perform the copy in Finder or on the command line, not using export in Lr.
  3. In Lr, create a totally new catalog (File…New Catalog), also in the disposable directory, e.g., ~/Downloads/test/test-catalog/
  4. Import (add) the copied files into the new catalog, then launch NLP, convert the copied files, and apply NLP settings.
  5. Do you get exclamation marks?
  6. Reset the files in NLP.
  7. Change the catalog settings (Lightroom Classic…Catalog settings…Metadata) to enable “Automatically write changes into XMP”. (This won’t change the settings in your main catalog, just in the new one.)
  8. Launch NLP again, make some changes, and apply NLP settings.
  9. Do you still get exclamation marks? Do they appear and then disappear?

You can of course delete the new catalog after the test and reload your usual catalog. The result might help us to determine whether the error is due to a catalog/DB versus XMP mismatch, or whether it’s something different.

9

Is automatic writing of xmp what resolves the issue on your Mac?

10

Yes, but it’s only a partial hypothesis because it doesn’t really explain why the exclamation marks appear in the first place (but see my reasoning, below). It’d be great to have independent confirmation from someone who has the same (or a very similar) problem. In my case, automatically writing into XMP does not circumvent the exclamation marks (and so the error), but it does make them disappear.

My thinking is as follows, but I’m looking for confirmation: NLP writes metadata to the Lr catalog, but it doesn’t modify proprietary RAW files (ARWs, CR2s, etc.) or their associated XMP files by default. So with these files, when there’s no associated sidecar file, there’s no error. The error occurs either with DNG files, which have XMP metadata baked in, or with RAW files that have sidecar XMP files (as in your case). So what’s the source of the error/exclamation marks? I think it must be that the metadata (correctly) written to the Lr catalog is no longer identical to the DNG metadata (or sidecar XMP metadata, in your case), which presents as a write error. If this thinking is correct, then, in your case as in mine, enabling automatic writing into XMP should also make the exclamation marks disappear as soon as they appear. Would it be possible for you to check this with a new catalog?

(As a more general point, I personally don’t think there’s much of a downside to writing metadata to DNGs or XMP sidecar files. Maybe people have an opinion on this. Some people think it causes a slight performance hit, but I don’t think that should be much of a problem on recent machines, if it’s a problem at all. The upside is that all the editing metadata from the Lr catalog is also written into XMP, which serves as a sort of backup of your editing history. If your catalog ever gets corrupted, you can reload your editing history easily into a new catalog.)

11

Just adding that I have now tried this on a completely different Mac and I’m seeing the same problem.

System notes (same as for the other machine):
Negative Lab Pro 3.0.2
“Negative Lab - Sony ILCE-6700 v2.3.dcp”
Lightroom Classic 13.1
MacOS Sonoma 14.2.1

12

I think I may just go back to using my ARWs, in case there’s a problem with other scenarios. But perhaps @nate could please take a look at this thread and let us know if there’s anything we can do? Thanks to all.

14

@iimd ,it’s certainly a good idea for @nate to look into this issue. I still suspect that access rights are part of the problem. Folder and file permissions depend on how a folder was created (by the Finder or by Lightroom upon importing) and how the files were added to the folder. The location of the archive might be important too. I keep it in the shared user folder (because I was using several user accounts and wanted easier access) vs the standard location in the logged-in user’s Pictures folder.

  • folder created with Finder or Lightroom
  • files added by Finder or Lightroom
  • location of photo archive: home vs shared
  • user account: admin or non-admin
  • logged-in user initiating the actions in relation to aforementioned users involved.

Note: user in the sense of user account used above.

15

I ran further tests with some randomly downloaded RAW images from 3 different cameras, all confirming the issue mentioned above. You can see the exclamation marks here:

Screenshot 2024-01-05 at 2.46.34 AM
Screenshot 2024-01-05 at 2.46.34 AM3104×1974 496 KB

(I used some positive images for convenience and source variety, but I don’t think that matters for the purposes of this discussion.)

However, here is some new information: I also started a new catalog with “Automatically write changes into XMP” enabled. I added the RAW files and saved the XMPs. I then merely launched NLP (without converting anything), watched the exclamation marks appear and disappear, and then again saved the modified XMP files. I then diff’ed the before and after XMP files. Sample output:

$ diff ./xmp1/20200703-DSC00483.xmp ./xmp2/20200703-DSC00483.xmp
19c19
<    xmp:MetadataDate="2024-01-05T02:38:58-05:00"
---
>    xmp:MetadataDate="2024-01-05T03:05:29-05:00"
74c74
<    xmpMM:InstanceID="xmp.iid:843243aa-e57b-4617-a31e-d714c0a95604"
---
>    xmpMM:InstanceID="xmp.iid:03b7bc5a-754f-4194-9664-ae0400a34bac"
205a206,211
>       stEvt:softwareAgent="Adobe Photoshop Lightroom Classic 13.1 (Macintosh)"
>       stEvt:changed="/metadata"/>
>      <rdf:li
>       stEvt:action="saved"
>       stEvt:instanceID="xmp.iid:03b7bc5a-754f-4194-9664-ae0400a34bac"
>       stEvt:when="2024-01-05T03:05:29-05:00"

Nothing very important. But of course after an NLP conversion, there were a ton of differences.

I am still wondering whether this issue is perhaps due to the fact that NLP just doesn’t write the metadata to elsewhere than the catalog. But of course Lr does write the metadata to elsewhere than the catalog, but only if you tell it to, e.g., by enabling “Automatically write changes into XMP”.

But what confuses me is that it seems to me that DNG files should by rights contain all the XMP metadata by default, otherwise what’s the good of having a DNG file — that by the way seems to cause an error in Lr if it doesn’t match what’s in the Lr catalog? And if a user like @Digitizer already has XMP files, which maybe come from some other source, then the different metadata locations could also, I suppose, confuse Lr. Maybe all of this just comes down to the fact that Lr just wants us to decide on our own where to put the metadata. In which case, the exclamation mark would just mean: “metadata not saved outside the catalog”. If that’s all it is, then maybe NLP could provide the user with some explanation regarding the “error writing metadata” and what it implies, especially if it’s not an important error. Because it certainly looks important when you see all those exclamation marks. It can make it look like there’s a disk write error or a permissions problem.

Anyway, I’m speculating. And hoping that the error is not important. It would be super if someone could shed some light on this issue soon.

16

Quite right, especially with recent MacOS security enhancements. I tried to rule out this sort of issue by giving Lr full disk access in “System Settings…Privacy & Security…Full Disk Access”.

17

Come to think of it, I don’t think Lr writes even the most mundane metadata tags to DNGs, unless you explicitly tell it to (via Cmd-S or “Automatically write metadata into XMP”). So if you change the Creator tag in Lr with the Creator tag already set to something else in the DNG, then Lr doesn’t automatically update the DNG. It waits for you to hit Cmd-S. On the other hand, this sort of mismatch between catalog and XMP written into the DNG does not cause an exclamation mark.

18

I don’t know if this is relevant, but I used exiftool to look at the metadata in an ARW and then in the corresponding DNG (post-DNG and post-NLP conversions). There is an “Error reading SR2 data” in the DNG that is not in the ARW. Also, there is some metadata in the ARW that seems to have gone missing in the DNG.

$ exiftool -s -a -G1 A43_19690530_01.dng | grep -i "sr2"
[ExifTool]      Warning                         : [minor] Bad offset for AdobeSR2 MRWInfo
[ExifTool]      Warning                         : Error reading SR2 data
[SR2]           SR2SubIFDOffset                 : 81498
[SR2]           SR2SubIFDLength                 : 35552
[SR2]           SR2SubIFDKey                    : 0x44332211

And now for the ARW file:

$ exiftool -s -a -G1 A43_19690530_01.arw | grep -i "sr2"
[SR2]           SR2SubIFDOffset                 : 81498
[SR2]           SR2SubIFDLength                 : 35552
[SR2]           SR2SubIFDKey                    : 0x44332211
[SR2SubIFD]     BlackLevel                      : 512 512 512 512
[SR2SubIFD]     WB_RGGBLevelsAuto               : 2612 1024 1024 1647
[SR2SubIFD]     WB_RGGBLevels                   : 2591 1024 1024 1629
[SR2SubIFD]     ColorMatrix                     : 1180 -169 13 146 1022 -145 82 -46 988
[SR2SubIFD]     WB_RGBLevelsDaylight            : 2605 1024 1659
[SR2SubIFD]     WB_RGBLevelsCloudy              : 2823 1024 1521
[SR2SubIFD]     WB_RGBLevelsTungsten            : 1586 1024 2996
[SR2SubIFD]     WB_RGBLevelsFlash               : 2820 1024 1483
[SR2SubIFD]     WB_RGBLevels4500K               : 2375 1024 1806
[SR2SubIFD]     WB_RGBLevelsShade               : 3080 1024 1345
[SR2SubIFD]     WB_RGBLevelsFluorescent         : 2401 1024 2242
[SR2SubIFD]     WB_RGBLevelsFluorescentP1       : 2593 1024 1639
[SR2SubIFD]     WB_RGBLevelsFluorescentP2       : 2878 1024 1525
[SR2SubIFD]     WB_RGBLevelsFluorescentM1       : 1843 1024 2926
[SR2SubIFD]     WB_RGBLevels8500K               : 3083 1024 1253
[SR2SubIFD]     WB_RGBLevels6000K               : 2763 1024 1496
[SR2SubIFD]     WB_RGBLevels3200K               : 1835 1024 2479
[SR2SubIFD]     WB_RGBLevels2500K               : 1439 1024 3403
[SR2SubIFD]     WhiteLevel                      : 15360 15360 15360
[SR2SubIFD]     VignettingCorrParams            : 11 0 32 112 248 432 672 952 1280 1648 2056 2488 0 0 0 0 0
[SR2SubIFD]     ChromaticAberrationCorrParams   : 22 256 512 640 768 768 768 640 640 512 512 384 -896 -768 -768 -768 -640 -640 -640 -512 -512 -384 -256 0 0 0 0 0 0 0 0 0 0
[SR2SubIFD]     DistortionCorrParams            : 11 -3 -1 2 5 11 16 24 33 45 58 74 91 111 132 157 185
[SR2DataIFD]    ColorMode                       : Standard
[SR2DataIFD1]   ColorMode                       : Vivid
[SR2DataIFD2]   ColorMode                       : Neutral
[SR2DataIFD3]   ColorMode                       : Portrait
[SR2DataIFD4]   ColorMode                       : FL
[SR2DataIFD5]   ColorMode                       : VV2
[SR2DataIFD6]   ColorMode                       : IN
[SR2DataIFD7]   ColorMode                       : SH
[SR2DataIFD8]   ColorMode                       : BW
[SR2DataIFD9]   ColorMode                       : Sepia

As you can see, there is no error reported in the ARW, but there is one in the DNG.

19

ExifTool can do a lot of things, some of which rely on reverse engineering undocumented file formats. I 'd advise to cautiously interpret ExifTool’s output.

Anyways, my guess is that “error writing metadata” is caused by some timing overlap that can occur with asynchronously writing metadata to xmp.

20

That sounds like an interesting hypothesis. How can we test it?

In the meantime, I diff’ed the exif metadata from two DNGs — one from before the presence of an exclamation mark and one after it appeared (when launching NLP but without converting anything). This is a little different from the previous diff results, which were from a comparison between an ARW and its DNG conversion. I again used exiftool -s -a -G1 filename.dng

Bearing in mind @Digitizer’s previous caution, here are the results:

4c4
< [System]        FileName                        : A00_19690101_12.dng
---
> [System]        FileName                        : A01_19690101_12.dng
7c7
< [System]        FileModifyDate                  : 2024:01:07 17:37:04-05:00
---
> [System]        FileModifyDate                  : 2024:01:07 17:37:47-05:00
9c9
< [System]        FileInodeChangeDate             : 2024:01:07 17:37:04-05:00
---
> [System]        FileInodeChangeDate             : 2024:01:07 17:37:47-05:00
174c174
< [XMP-xmp]       MetadataDate                    : 2024:01:07 17:36:23-05:00
---
> [XMP-xmp]       MetadataDate                    : 2024:01:07 17:37:47-05:00
184c184
< [XMP-xmpMM]     InstanceID                      : xmp.iid:d5a4b6f7-bdb1-470f-b0be-0a84ccbc6e71
---
> [XMP-xmpMM]     InstanceID                      : xmp.iid:14b96a65-0fdc-4adc-9d34-ede2786e2158
187,188c187,188
< [XMP-xmpMM]     HistoryInstanceID               : xmp.iid:7e8fd550-5445-4e81-a9e3-b7feef617a44, xmp.iid:d5a4b6f7-bdb1-470f-b0be-0a84ccbc6e71
< [XMP-xmpMM]     HistoryWhen                     : 2024:01:07 17:22:47-05:00, 2024:01:07 17:36:23-05:00
---
> [XMP-xmpMM]     HistoryInstanceID               : xmp.iid:7e8fd550-5445-4e81-a9e3-b7feef617a44, xmp.iid:14b96a65-0fdc-4adc-9d34-ede2786e2158
> [XMP-xmpMM]     HistoryWhen                     : 2024:01:07 17:22:47-05:00, 2024:01:07 17:37:47-05:00

From the point of view of exiftool, that’s the (rather uninteresting) content that corresponds to the exclamation mark. There may be other differences that exiftool does not report.

As usual, clicking on the exclamation mark in Lr and selecting “Retry Metadata Export” seems to clear the error.

21

Different filenames, timestamps and IDs, nothing exciting as you commented indeed.

I have been moving database backups and/or photo archives or both from time machine backups and restarted Lightroom with these odd and even couples and found that exclamation marks occasionally appear and go away automatically or not, even though Lr is set to not write xmp automatically. This temporary presence of warning badges leads me to suspect that one of Lightroom’s threads lights the warnings (by e.g. simply checking file(system) change timestamps, while an other thread has not yet compared the sidecar file contents to the database properly, but that later turns the warning off when the inspection is done. This could happen with changes done in Lightroom or the underlying OS in combination with so-so design or testing.

Note that I still use Lr version 12 on macOS Monterey, both in their latest published releases.