Requesting a proper iSRD pipeline

Feature Requests
1

Hello, Negative Lab Pro creator(s)!

I’ve just bought Negative Lab Pro after some tests. I really liked the product. But there is one huge drawback – since I scan everything in DNG 64 HDRi RAW the pipeline is too complex when working with files containing an infrared channel for dust removal.

Your team is really cool, you wrote such a useful and high-quality plugin, but if the user of your products also owns SilverFast HDR Studio, then using the dust removal feature is very difficult and requires many additional steps which I read about on this forum. Therefore, my feature request is as follows. Since you are obviously talented in programming, I think it won’t be difficult for you to add functionality to this plugin that, for those DNG(/TIFF?) container files created in SilverFast and containing an infrared channel, will extract this channel, for example, into a separate file (for example after processing by the NegativeLab plugin). Then, literally in one click, this mask file with information about dust and scratches can be “subtracted” from the final image in Photoshop. For example, using the “Content-Aware fill” function. I would even be ready to pay money for this feature if it exist. I think that such feature is literally indispensable.
.

PS: And I think many users will be happy to be able to use Photoshop’s algorithms to remove dust_and_scratches, because the algorithms that SilverFast uses for post-processing these things are quite poor.

2 Likes
2

I would also be very interested in this functionality, and I think it shouldn’t be too hard either! Would love to pay for this :slight_smile:

I haven’t purchased yet, but I am trying to figure out the right workflow for use with my Plustek 8200i Ai for flexibility.

It seems there are three options from what I’ve read:

  • Go with the “Silverfast RAW DNGs” as described in the guide and scan 48 bit HDR RAW: Film Scanning Best Practices | Negative Lab Pro - but then I am stuck without any iSRD dust removal.

  • Scan to TIFF - I tried doing this at Gamma 1.0 but the colours were way off. I haven’t tried it yet at Gamma 2.2 but I read posts where someone said that the colours were better there. I also saw a post on the NLP Facebook group where someone compared DNG to TIFF scans processed with NLP and I personally thought the DNG results looked better as well. I don’t know why this is (I’m new to all of this) but I would be interested in why that might be.

  • Scan to 64 bit HDRi RAW with the infrared channel included, but I don’t think there is a reasonable workflow for this other than using the Silverfast software. And I would really like to use NLP if possible, as I like the results others are getting.

Any thoughts? Should I just scan as 64 bit HDRi going forward and figure out how to process the files later?

Thank you for any advice, and would love to see the suggestion @galxcom proposed come to fruition if possible!

2 Likes
3

It sounds here like this is something Nate might already be aware of and possibly working on: Current recommended SilverFast settings? - #6 by nate

He mentioned:

I’m trying to find a better way to do this… the Tiff Scan Prep utility is not ideal and there are just too many caveats to it. I’m working on a method that could avoid this, and that could make for a better workflow for those wanting the IR dust removal embedded into their Silverfast scans.

1 Like
4

Apologies for the long post, but even though Negative Lab Pro doesn’t yet support this workflow, I would like to figure out some way of doing it on my own for now, so that I can do three things:

  1. Scan photos in the highest “archival” quality possible so they can be used with Negative Lab Pro.
  2. Capture the IR data using my scanner while doing my scans so I can process them to remove dust.
  3. Figure out a way to apply iSRD using Photoshop later on using the IR data.

So I did some testing on how I could scan images as 64 bit HDRi in Silverfast or Vuescan or any other software and then later on somehow batch process them with some sort of Photoshop macro to use content-aware fill to remove the dust and scratches using an extracted 16 bit IR channel.

I am not an expert on this but I figured I would do some research and take a stab at it!

I saw online that you can use imagemagick for this sort of task, so I gave it a shot and installed it:

connor@Connors-MacBook-Air magick % brew install imagemagick
connor@Connors-MacBook-Air magick % brew install ufraw

I did two 64 bit scans, in DNG and TIFF formats, and then used the Imagemagick identify command to list out what the files contain. I was hoping Imagemagick would see the IR channel in the DNG file so I could extract the IR channel to a separate file and use it later. I want to use DNG rather than TIFF as Negative Lab Pro seems to like these files better.

Let’s find out what identify sees on the files:

connor@Connors-MacBook-Air magick % identify -verbose 35mm05-07.dng 
Image:
  Filename: /var/folders/dx/6dr1gbhx6mv4zk9tp5lyqf4h0000gn/T/magick-ckn3tksKCx_0lgYQ5s_uRI8SknjGdVdv.png
  Base filename: 35mm05-07.dng
  Format: DNG (Digital Negative)
  Class: DirectClass
  Geometry: 3528x5184+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 18289152
    Red:
      min: 0  (0)
      max: 55009 (0.839384)
      mean: 37990.6 (0.5797)
      median: 292 (0.00445563)
      standard deviation: 11179.4 (0.170587)
      kurtosis: 3.10183
      skewness: -1.39089
      entropy: 0.922597
    Green:
      min: 0  (0)
      max: 61444 (0.937575)
      mean: 36145.6 (0.551547)
      median: 287 (0.00437934)
      standard deviation: 15241.9 (0.232577)
      kurtosis: -0.418933
      skewness: -0.550743
      entropy: 0.965406
    Blue:
      min: 0  (0)
      max: 56295 (0.859007)
      mean: 37118.8 (0.566396)
      median: 421 (0.00642405)
      standard deviation: 13243.1 (0.202077)
      kurtosis: 0.425158
      skewness: -0.90535
      entropy: 0.952938
  Image statistics:
    Overall:
      min: 0  (0)
      max: 61444 (0.937575)
      mean: 37085 (0.565881)
      median: 333.333 (0.00508634)
      standard deviation: 13221.5 (0.201747)
      kurtosis: 0.590982
      skewness: -0.882456
      entropy: 0.94698
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 3528x5184+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Zip
  Orientation: Undefined
  Properties:
    date:create: 2021-05-17T18:53:56+00:00
    date:modify: 2021-05-17T18:53:56+00:00
    dng:Aperture: F/0.0
    dng:AspectRatio: 0.680556
    dng:ChannelMultipliers: 1.000000 2.269203 2.972209 1.000000
    dng:CreateID: 1
    dng:Crop: 0 0 3528 5184
    dng:EXIFSource: DCRaw
    dng:FocalLength: 0.0 mm
    dng:Green: 1.495
    dng:ISOSpeed: 0
    dng:Make: Plustek
    dng:MatrixInputProfile: Color matrix
    dng:Model: OpticFilm 8200i
    dng:Orientation: 0
    dng:Rotation: 0.000000
    dng:Shutter: 0.0 s
    dng:sRGBOutputProfile: sRGB
    dng:Temperature: 3912
    dng:Timestamp: Thu Jan  1 00:00:00 1970
    dng:WB: Auto WB
    dng:WBFineTuning: 0
    png:IHDR.bit-depth-orig: 16
    png:IHDR.bit_depth: 16
    png:IHDR.color-type-orig: 2
    png:IHDR.color_type: 2 (Truecolor)
    png:IHDR.interlace_method: 0 (Not interlaced)
    png:IHDR.width,height: 3528, 5184
    png:sRGB: intent=0 (Perceptual Intent)
    png:text: 2 tEXt/zTXt/iTXt chunks were found
    signature: 03d4c4d5a6b34fa9af6f65a8ddb068ff2c448c660bb1ebb91cfce77c1bd22c33
    Software: UFRaw
    Source: PlustekOpticFilm 8200i
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 85.7493MiB
  Number pixels: 18.2892M
  Pixels per second: 26.5656MP
  User time: 0.690u
  Elapsed time: 0:01.688
  Version: ImageMagick 7.0.11-12 Q16 arm 2021-05-09 https://imagemagick.org

We can see here that there are 3 channels on the DNG - but I want to find the IR (alpha?) channel and it doesn’t appear to be visible to Imagemagick from the DNG, unless I’m missing something.

connor@Connors-MacBook-Air magick % identify -verbose 35mm05-07.tif 
Image:
  Filename: 35mm05-07.tif
  Format: TIFF (Tagged Image File Format)
  Mime type: image/tiff
  Class: DirectClass
  Geometry: 3528x5184+0+0
  Resolution: 300x300
  Print size: 11.76x17.28
  Units: PixelsPerInch
  Colorspace: sRGB
  Type: TrueColor
  Endianness: MSB
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 18289152
    Red:
      min: 0  (0)
      max: 44011 (0.671565)
      mean: 22855.3 (0.34875)
      median: 114 (0.00173953)
      standard deviation: 10208.5 (0.155772)
      kurtosis: -0.452543
      skewness: 0.116265
      entropy: 0.958449
    Green:
      min: 0  (0)
      max: 25110 (0.383154)
      mean: 10091 (0.153979)
      median: 48 (0.000732433)
      standard deviation: 6335.23 (0.0966695)
      kurtosis: -1.07491
      skewness: 0.329503
      entropy: 0.972197
    Blue:
      min: 0  (0)
      max: 15851 (0.241871)
      mean: 7690.9 (0.117356)
      median: 50 (0.000762951)
      standard deviation: 4050.22 (0.0618025)
      kurtosis: -1.10885
      skewness: -0.00485492
      entropy: 0.978211
  Image statistics:
    Overall:
      min: 0  (0)
      max: 44011 (0.671565)
      mean: 13545.7 (0.206695)
      median: 70.6667 (0.0010783)
      standard deviation: 6864.66 (0.104748)
      kurtosis: 0.618621
      skewness: 1.02529
      entropy: 0.969619
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 3528x5184+0+0
  Dispose: Undefined
  Iterations: 0
  Scene: 0 of 3
  Compression: None
  Orientation: TopLeft
  Profiles:
    Profile-xmp: 145316 bytes
  Properties:
    date:create: 2021-05-17T18:52:19+00:00
    date:modify: 2021-05-17T18:52:07+00:00
    exif:MakerNote: 0x4c,0x53,0x49,0x31,0x0,0xf,0xc0,0x2,0x0,0x1,0x0,0x0,0x0,0x3d,0xc0,0x9,0x0,0x1,0x0,0x0,0x20,0x8,0xc0,0x5,0x0,0x1,0x0,0x0,0x0,0x1,0xc0,0x7,0x0,0x1,0x0,0x0,0x0,0x1,0xc0,0x8,0x1,0x1,0x2,0x25,0x51,0x0,0xc0,0xf,0x0,0x1,0x0,0x0,0x0,0x1,0xc0,0x10,0x0,0x1,0x0,0x0,0x0,0x0,0xc0,0xa,0x0,0x1,0x0,0x0,0x0,0x5,0xc0,0x4,0x1,0x1,0x0,0x0,0x38,0x40,0xc0,0x3,0x1,0x1,0x0,0x0,0x26,0x44,0xc0,0xb,0x1,0x1,0x0,0x0,0x0,0x0,0xc0,0xc,0x1,0x1,0x0,0x0,0x0,0x30,0xc0,0xd,0x1,0x1,0x0,0x0,0x38,0x40,0xc0,0xe,0x1,0x1,0x0,0x0,0x27,0x10,0xc0,0x1,0x1,0x1,0x0,0x0,0x27,0x10,0x4b,0x49,0x45,0x4c
    exif:PixelXDimension: 3528
    exif:PixelYDimension: 5184
    signature: 708840bcf013fafbf7fc0239d01bda6163b92bb6ae49348170101aa8b3a21837
    tiff:alpha: unspecified
    tiff:endian: msb
    tiff:make: Plustek
    tiff:model: OpticFilm 8200i
    tiff:photometric: RGB
    tiff:rows-per-strip: 1
    tiff:software: SilverFast 8.8.0 r24 (Oct 29 2020)  30c8121 29.10. 
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 148.342MiB
  Number pixels: 18.2892M
  Pixels per second: 118.836MP
  User time: 0.150u
  Elapsed time: 0:01.153
  Version: ImageMagick 7.0.11-12 Q16 arm 2021-05-09 https://imagemagick.org
Image:
  Filename: 35mm05-07.tif
  Format: TIFF (Tagged Image File Format)
  Mime type: image/tiff
  Class: DirectClass
  Geometry: 1010x1484+0+0
  Resolution: 85.8844x85.8844
  Print size: 11.76x17.279
  Units: PixelsPerInch
  Colorspace: sRGB
  Type: TrueColor
  Endianness: MSB
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 1498840
    Red:
      min: 0  (0)
      max: 43472 (0.66334)
      mean: 22828.7 (0.348344)
      median: 114 (0.00173953)
      standard deviation: 10209.9 (0.155793)
      kurtosis: -0.455103
      skewness: 0.104322
      entropy: 0.959115
    Green:
      min: 0  (0)
      max: 23557 (0.359457)
      mean: 10079.3 (0.1538)
      median: 46 (0.000701915)
      standard deviation: 6319.33 (0.0964268)
      kurtosis: -1.10361
      skewness: 0.313815
      entropy: 0.971978
    Blue:
      min: 0  (0)
      max: 15172 (0.23151)
      mean: 7681.95 (0.117219)
      median: 52 (0.000793469)
      standard deviation: 4045.27 (0.0617268)
      kurtosis: -1.11303
      skewness: -0.014116
      entropy: 0.97654
  Image statistics:
    Overall:
      min: 0  (0)
      max: 43472 (0.66334)
      mean: 13530 (0.206454)
      median: 70.6667 (0.0010783)
      standard deviation: 6858.16 (0.104649)
      kurtosis: 0.607765
      skewness: 1.02153
      entropy: 0.969211
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 1010x1484+0+0
  Dispose: Undefined
  Iterations: 0
  Scene: 1 of 3
  Compression: None
  Orientation: TopLeft
  Properties:
    date:create: 2021-05-17T18:52:19+00:00
    date:modify: 2021-05-17T18:52:07+00:00
    signature: 57273bbb1aca54c0bfe189a4535112a1583f96ee5bb0fae6e0d547b4bdfffacf
    tiff:alpha: unspecified
    tiff:endian: msb
    tiff:photometric: RGB
    tiff:rows-per-strip: 1
    tiff:subfiletype: REDUCEDIMAGE
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 0B
  Number pixels: 1498840
  Pixels per second: 699357P
  User time: 3.190u
  Elapsed time: 0:03.143
  Version: ImageMagick 7.0.11-12 Q16 arm 2021-05-09 https://imagemagick.org
Image:
  Filename: 35mm05-07.tif
  Format: TIFF (Tagged Image File Format)
  Mime type: image/tiff
  Class: DirectClass
  Geometry: 3528x5184+0+0
  Resolution: 300x300
  Print size: 11.76x17.28
  Units: PixelsPerInch
  Colorspace: Gray
  Type: Grayscale
  Endianness: MSB
  Depth: 16-bit
  Channel depth:
    Gray: 16-bit
  Channel statistics:
    Pixels: 18289152
    Gray:
      min: 0  (0)
      max: 57673 (0.880034)
      mean: 49963.3 (0.76239)
      median: 0 (0)
      standard deviation: 11339.1 (0.173024)
      kurtosis: 13.5697
      skewness: -3.85923
      entropy: 0.81399
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 3528x5184+0+0
  Dispose: Undefined
  Iterations: 0
  Scene: 2 of 3
  Compression: None
  Orientation: TopLeft
  Properties:
    date:create: 2021-05-17T18:52:19+00:00
    date:modify: 2021-05-17T18:52:07+00:00
    signature: 7d2b606194fe833c9efa07212de8f84172e18a0650cf7905f34df6ac916d3d41
    tiff:alpha: unspecified
    tiff:endian: msb
    tiff:photometric: min-is-black
    tiff:rows-per-strip: 1
    tiff:subfiletype: MASK
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 0B
  Number pixels: 18.2892M
  Pixels per second: 7.91901MP
  User time: 3.540u
  Elapsed time: 0:03.309
  Version: ImageMagick 7.0.11-12 Q16 arm 2021-05-09 https://imagemagick.org
identify: Unknown field with tag 35243 (0x89ab) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/960.
identify: Unknown field with tag 35244 (0x89ac) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/960.
identify: Unknown field with tag 35244 (0x89ac) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/960.

Aha – so, it seems that with the TIFF, Imagemagick IS able to see what I believe is the second TIFF “page” that contains the IR channel (16-bit greyscale), but it can’t see it when the photo is in DNG format.

However, I was hoping that I could scan with DNG so that things would work best with NLP/LR, so this isn’t very encouraging. I’m not sure why it’s not visible when it’s a DNG, but to test my theory, I used Imagemagick to convert from DNG to TIFF, and sure enough the file size went down, so the photo seems to have lost data, which I assume is because Imagemagick isn’t doing anything with the Silverfast IR data that it isn’t seeing properly, and doesn’t know how to handle when processing a DNG (I guess DNG files don’t usually have an IR channel!).

-rw-r–r-- 1 connor staff 105M 17 May 14:58 35mm05-07-converted.tif
-rw-r–r-- 1 connor staff 149M 17 May 13:52 35mm05-07.dng
-rw-r–r-- 1 connor staff 148M 17 May 14:52 35mm05-07.tif

As you can see, the file is almost 43MB smaller than the Silverfast TIF after conversion from the Silverfast DNG. Let’s see what identify says about this new TIFF.

connor@Connors-MacBook-Air magick % identify -verbose 35mm05-07-converted.tif
Image:
  Filename: 35mm05-07-converted.tif
  Format: TIFF (Tagged Image File Format)
  Mime type: image/tiff
  Class: DirectClass
  Geometry: 3528x5184+0+0
  Units: PixelsPerInch
  Colorspace: sRGB
  Type: TrueColor
  Endianness: MSB
  Depth: 16-bit
  Channel depth:
    Red: 16-bit
    Green: 16-bit
    Blue: 16-bit
  Channel statistics:
    Pixels: 18289152
    Red:
      min: 0  (0)
      max: 55009 (0.839384)
      mean: 37990.6 (0.5797)
      median: 292 (0.00445563)
      standard deviation: 11179.4 (0.170587)
      kurtosis: 3.10183
      skewness: -1.39089
      entropy: 0.922597
    Green:
      min: 0  (0)
      max: 61444 (0.937575)
      mean: 36145.6 (0.551547)
      median: 287 (0.00437934)
      standard deviation: 15241.9 (0.232577)
      kurtosis: -0.418933
      skewness: -0.550743
      entropy: 0.965406
    Blue:
      min: 0  (0)
      max: 56295 (0.859007)
      mean: 37118.8 (0.566396)
      median: 421 (0.00642405)
      standard deviation: 13243.1 (0.202077)
      kurtosis: 0.425158
      skewness: -0.90535
      entropy: 0.952938
  Image statistics:
    Overall:
      min: 0  (0)
      max: 61444 (0.937575)
      mean: 37085 (0.565881)
      median: 333.333 (0.00508634)
      standard deviation: 13221.5 (0.201747)
      kurtosis: 0.590982
      skewness: -0.882456
      entropy: 0.94698
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)
  Matte color: grey74
  Background color: white
  Border color: srgb(223,223,223)
  Transparent color: none
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 3528x5184+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: None
  Orientation: TopLeft
  Properties:
    date:create: 2021-05-17T18:58:48+00:00
    date:modify: 2021-05-17T18:58:48+00:00
    signature: 03d4c4d5a6b34fa9af6f65a8ddb068ff2c448c660bb1ebb91cfce77c1bd22c33
    tiff:alpha: unspecified
    tiff:endian: msb
    tiff:photometric: RGB
    tiff:rows-per-strip: 48
  Artifacts:
    verbose: true
  Tainted: False
  Filesize: 104.652MiB
  Number pixels: 18.2892M
  Pixels per second: 55.5569MP
  User time: 0.110u
  Elapsed time: 0:01.329
  Version: ImageMagick 7.0.11-12 Q16 arm 2021-05-09 https://imagemagick.org

Sure enough, the second page is not present, so no IR data after converting from DNG > TIFF. We only have the RGB channels.

I did some googling and found a forum post about Vuescan referencing ImageJ, so I figured I would give it a try to see what it saw on the DNG vs the TIFF. I downloaded it from: Download and chose the M1 version for my M1 Macbook Air.

I followed the instructions on the first reply in this post: Create TIF with 4 channels per pixel? - Usage & Issues - Image.sc Forum

Basically I went to Image > Stacks > Stack to Images. This split out the channels.

Properties

11.76x17.30 inches (174x25

However, it seems that ImageJ, like Imagemagick also only sees the RGB channels on the DNG, and in fact when I tried it on the Silverfast TIFF it didn’t split out the IR to a stack either. So I guess that test wasn’t very useful. But I’m including it here so you guys can see what I tried.

So at this point I’m a bit stumped. What’s so special about this DNG file that Imagemagick can’t see the IR data? Also, am I losing any data / colour information by scanning as TIFF rather than DNG? Basically I want to scan in the format which will give me the most flexibility but without sacrificing any colour info. If I can get away with scanning to TIFF and being able to use my IR data, then that would probably be the best option I reckon.

I did some more googling to confirm if this is true (I’m sorry, this is all new to me so I am trying to do my homework but I’m sure you guys can correct me) and found a forum post which seemed to confirm that it’s just a container format (https://www.rangefinderforum.com/forums/showthread.php?t=117590):

To start with, the difference between a digital camera raw file and a Vuescan raw file is that scanner files are ‘linear raw files’ while camera files are mosaiced (Sigma cameras being the exception here). A scanner file is basically an RGB TIFF file (three color channels) while a camera file is a single monochrome channel with proprietary (or standard if it is a DNG) de-mosaicing information tagged onto it. Hence the difference in file size between one and three channels.

In all other respects they are very similar. A raw file is a TIFF file with no, or minimal adjustments applied to it. A raw file will have no color profile, and be gamma 1.0. This is why it is so dark. When you import a raw file into ACR or Lightroom, the first thing it does is apply a generic gamma base curve to it to lighten it up (making it 1.8 or 2.2 i believe). So when you open it up and set all the sliders to 0, it is not really 0, because it assumes that you want to increase the gamma. When you export a regular TIFF (non-raw) from Vuescan it will apply a gamma curve and a color profile. If you would import this file into Lightroom, it will appear very high contrast because it is getting a second base curve applied to it.

A linear DNG file is a TIFF file with other EXIF information tagged onto it. The raw files that Vuescan exports is identical regardless of whether the ending is DNG or TIFF. You can also reprocess them in in Vuescan without the format making any difference. I think it is important to note that raw does not necessarily mean that nor processing has been applied to a file. My workflow is that I typically scan and output a 64bit RGBI TIFF, which I reprocess in order to apply the dust filter, and then I output a 48 bit DNG file for Lightroom. I also let Vuescan invert negatives which I still export as raw. Hens the output from ‘Scan’ or ‘Save’, and the ‘Raw save film’ option (confusingly named unfortunately)

The logic to outputting a DNG as opposed to a TIFF for use in Lightroom is firstly just the convenience of not having the sidecar files in Lightroom (the DNG format embeds the settings in the file). The most important difference is in the Camera Calibration pane in Lightroom. When you import a camera raw file, Lightroom recognizes the camera and offers color profiles specific to your camera sensor. When you import a generic TIFF, it will apply default settings that you have no control over. With a DNG file from Lightroom it will recognize your scanner as a ‘camera’. Using the DNG Profile Editor (free from adobe) you will be able to make custom scan profiles for your scanner and apply them to different film types. For instance I have profiles that have a base curve that I use with negative scans, and ones without base curve for scanned slides because slides are ‘gamma correct’ already.

So if this guy is correct, and the DNG really doesn’t have any secret colour info stored in it (what does the extra EXIF metadata in a DNG contain then??), and the advantages of using DNG with Negative Lab Pro are only because of how Lightroom handles DNGs, then I think I should probably scan as 64 bit iSRD TIFF, process the files for IR by splitting our the alpha/IR channel and running it through photoshop somehow (not sure the logistics of this but I am sure it must be possible), and then prepare the processed TIFF for LR by converting it back to a DNG, so that it will work nicely with NLP and how Lightroom handles gamma?

Let me know your thoughts everyone!

Thanks,

Connor

EDIT, added on more to this post:

I hope my giant additions to this topic are alright as I realize this is a “feature request” forum and while I would love the feature, my current concern is more around figuring out a workflow in the meantime — please let me know if I’m breaking any rules :slight_smile: - I can create a new thread instead, if preferred.

I found an interesting discussion thread on Flickr from 2011 which reveals that, apparently, Silverfast pre-V8 supported iSRD with Linear scans but that feature was removed:

As some of us have discovered, Lasersoft has updated their SilverFast software from v.6.6 to v.8.

Among the new features, the possibility to use dust and scratch removal (SRD and iSRD) while saving linear scans is now removed. According to LaserSoft’s support team: “In case you want to make any changes to your image (including iSRD) please use the normal 48bit mode”

The consequence of this is that if you want to use SilverFast scanning software and ColorPerfect, you’ll have to go back to the old days before infrared dust and scratch removal and use the clone tool in Photoshop instead.

Or you could switch to VueScan.

Why? I can only guess that LaserSoft wants to limit their customers’ choice of tools and force them to use LaseSoft’s “HDR suite” instead of their favorite software.

The OP’s conclusion:

You don’t lose anything by saving a DNG, in fact you gain some additional metadata that your raw developer uses as an input colour profile. If you later want to convert to a raw (or rendered) TIFF or any other format you can use VueScan to “scan” your DNG file into those, or you can use a tool like dcraw to do it.

At this point I think maybe I should try and experiment a bit with Vuescan to see if I can get acceptable results scanning to DNG with the infrared data included. Maybe it will be easier to work with than Silverfast, and I can try applying the iSRD there.

I found another thread on Flickr called “How to apply the infrared channel to the image in PhotoShop?” and he said “Vuescan’s RAW files are meant to be reprocessed in Vuescan itself, not Photoshop.”, so I guess I would be stuck using Vuescan.

Like @galxcom mentioned in his OP, I would really love to see if Photoshop’s algorithms (content aware fill?) do a nicer job of removing dust than Vuescan or Silverfast do. I know that Silverfast does a good job, but it’s not exactly “pretty” if you are pixel peeping. You can easily tell that an edit has been done, it looks kind of smudgey.

5

The search work you’ve done is really impressive! Thank you for writing in detail in this thread. I myself have done almost all the operations you mentioned with magick and ImageJ before, but I did not write here, because all those tryouts did not lead to success, and my initial text was already too big for a “feature request”. Plus, my LightRoom, for some reason, often couldn’t add the aforementioned TIFF files to its catalog, while adding DNG files to its catalog has never been a problem. Therefore, the SilverFast HDRi TIFFs → … pipeline was not considered by me. I considered only pipeline SilverFast HDRi DNGs → LR + NegativeLabPro → Photoshop action [Dust remove with content aware fill + NIK Sharpener pro (which is free)].

NegativeLabPro’s goal is to do everything with one program/click if I understood correctly. Thus NegativeLabPro dev(s) could really say for us and this idea “bye” but who knows. Since I was a UI motion graphic artist for me it would me easy to offer this idea, @nate, take a look:

Снимок экрана 2021-05-17 в 21.59.14
Снимок экрана 2021-05-17 в 21.59.14860×1706 83 KB

And here we go! Digital Negative (DNG) SpecificationVersion 1.5.0.0May 2019:
https://wwwimages2.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.5.0.0.pdf so for such professional developer(s) of NegativeLabPro i bet would be really easy to get into this format and implement the option of extracting that damn IR pass…

Regards!

1 Like
6

So a quick update for my situation, I just ended up purchasing a license for Vuescan as well as Negative Lab Pro! :confetti_ball:

I want to continue scanning, but I would like to be able to use my files in NLP (I want to make sure they are 48bit HDR DNG files so they will be compatible - I tried using 64 bit HDRi and as expected from the docs, they wouldn’t work with Lightroom/NLP).

I got Vuescan, since it allows you to do iSRD and keep it with your scan or as a separate file, and while it’s not as aggressive as Silverfast’s correction was, it seems to be adequate for me and I think I might be able to find a way to use the infrared scan in the future to do further dust removal.

Under the filter tab I set Vuescan to do “medium” iSRD. It seems to catch the “worst” of the dust but there are still plenty of tiny specks or micro scratches left which could be fixed through the IR I think (depends on the negative and how damaged/dirty it is, of course). This iSRD gets baked into the DNG output with the settings I’ve chosen.

Under the output tab I have got it creating two files:

  • RAW DNG for Lightroom / NLP. This has Vuescan’s “medium” iSRD baked in, which I find to be not very aggressive.
  • 16 bit compressed TIFF with the infrared channel only so I can possibly use this later on and process the files even further for dust. I’m not sure entirely how I could utilize it, but there has to be a way. I found this thread, and there are some scripts there which might be useful: Scanned image scratch removal with “ICE” - Processing - discuss.pixls.us, so I’ll have to do more research. Would be great if I could just batch process my photos for dust with a script or something.

I’m really happy with the results so far! The colours look way better even on this old 80s Agfa negative which seems to have a acquired a pretty intense green cast for some reason.

Original Silverfast JPG Scan with Negafix and iSRD

35mm01-12
35mm01-125050×3406 2.41 MB

Vuescan, Medium iSRD, RAW DNG processed with Negative Lab Pro

2021-05-17-0009-2
2021-05-17-0009-25045×3406 2.49 MB

Maybe I actually don’t even really mind the small extra pieces of dust, it almost looks more natural, but it’s nice to know I can try and correct them later I guess. Perhaps I should just not bother with it though. The scan looks pretty good already.

Infrared Channel (Converted from the TIFF to JPG to be able to upload it here)

2021-05-17-0009-raw
2021-05-17-0009-raw5184×3518 563 KB

Would still love to just be able to scan from Silverfast or Vuescan to those 64 HDRi files and be able to throw them into NLP though, because that would remove any guessing from the process! But, for now this is my workaround. And hopefully I can find a way to do dust removal from the command line using the

1 Like
7

@mcbrineellis it looks like your text is unfinished – it breaks off at the end.

I would still like to be able to extract the infrared channel from 64 bit HDRi. Everything else is half measures. I spent about 800 euros to buy and upgrade software from SilverFast. Plus, $ 120 for NLP. And I understand that I am still not happy with the pipeline due to the lack of the possibility of normal automated cleaning of the frame from dust. However, I bet that for people who know how to program, the issue of extracting a channel from a DNG negative is extremely simple.

8

That’s a good idea, but I don’t think this will work quite like you expect for a couple of reasons…

First, I’m pretty sure that ImageMagick in this case will be using the ufraw delegate to create a temporary PNG, and then identify evaluates that PNG (not the original DNG).

You can see evidence of this in the first line you get back…

Filename: /var/folders/dx/6dr1gbhx6mv4zk9tp5lyqf4h0000gn/T/magick-ckn3tksKCx_0lgYQ5s_uRI8SknjGdVdv.png
Base filename: 35mm05-07.dng

The second reason that this may not work as you think is that there is more than one place that the IR “layer” could be stored. It could be stored as a channel layer, (and indeed, I believe that is how Vuescan stores the IR layer in their 64bit output).

But in the case of Silverfast, I believe they store the IR information as a LAYER and not as a channel. So if you are trying to separate the channels, you will not find it. You need to find the layer on which it is stored and extract it there.

I would see if you can find layer this via Exiftool (or if you have a 64bit DNG or 64bit TIF from Silverfast, I can also poke around). It’s possible that Silverfast is using their own special tag for this layer information, so it may take a bit of digging.

This is partially true, but doesn’t tell the whole story.

The information in the DNG is “scene-referred”, whereas the information in the TIF is display-referred. Basically, this means that the DNG information is provided relative to a given illumination. That’s why when you load a DNG, it shows the white balance as Kelvin (i.e. 6500k). And when you load a TIF, it just shows the white balance temperature as 0. This distinction actually lets us do a lot of useful stuff when it comes to processing negatives, as it gives me control via the camera profiles for what is happening behind the scenes when you white-balance off the film negative. The Tiff is more just “fixed” finished data, so it is less adjustable.

Thanks @galxcom - this has me thinking that maybe I could add something like this to the “Update Vuescan/Silverfast” utility that is already included… for instance, perhaps I could simply have it detect if the IR channel is present, and give users the option to extract it or even apply it to a new file.

2 Likes
9

This would be the ultimate dream, @nate. If you implement the function of extracting the IR channel in the utility that you mentioned, then I’ll definitely recommend the center of arts where I’ve been studying photography already for 3 years to purchase NLP for all computers, because this will seriously upgrade the working process. Every student here complains about absence of possibility of having fast negative editing software with high-quality dust removal option. This option will be if there be possibility to have a separated IR pass file.

10

www.dropbox.com/s/qbqioksqflokjcx/Test%20HDRi.zip?dl=0 – I uploaded to here an example from SilverFast in both formats DNG 64Bit HDRi and same TIFF. If this is still valid – feel free to use it.

11

Thanks @galxcom!

Yep, the IR data is stored as a part of a multi-page TIFF…

On the TIFF file itself, you can see this when you run identify -verbose and see three separate image tags:

  1. The full scan (4571x2983, RGB)
  2. The embedded preview (1523x994, RGB)
  3. The IR Channel (4571x2983, Gray)

You can split these out using the command: convert filename.tif filename%d.tif

The DNG file contains:

  1. The DNG data
  2. A tiff thumbnail (RGB)
  3. A tiff preview (RGB)
  4. A tiff IR channel (Gray)

You won’t see these when you run identify because of the way that imagemagick is working internally. It is just breaking off the DNG data using UFRAW and then evaluating a temporary PNG file it creates…

So if you want to break off the TIFFs embedded in the DNG, the easiest way is to run the command:

tiffsplit filename.dng

You will then see three new tiff files created, including the tiff with the IR data.
Screen Shot 2021-05-20 at 2.03.10 PM

The last file there is the IR tiff:
Screen Shot 2021-05-20 at 2.04.14 PM

The original DNG will remain intact. And incidentally, since Silverfast saves the IR data as a separate TIF layer and not as an alpha channel, the good news is that 64bit DNGs will run just fine in Lightroom and Negative Lab Pro (the same is not true for 64bit Vuescan DNGs since the IR layer is stored as a channel, and Lightroom is meant to work with 3-channel RAW files - although Vuescan has the advantage of saving as a 48bit DNG with the IR cleaning already factored in).

For the time being, you should then be able to process the DNG as normal in Negative Lab Pro, and then export the full, uncropped conversion to Photoshop, where you can add the IR layer and edit from there… I know there are some actions in Photoshop for dust removal with IR data, but they will probably need to be adapted a bit… but it’s a start…

1 Like
12

@nate , that’s amazing. Worked like a charm.
I’ve just tested for the first time my photoshop dust removal process. Previously I always did this in SilverFast HDR Studio. It has a quite bad algorithm for dust removal but it makes it very fast.

In photoshop i tried this sequence (and it worked perfectly but very slow):

Import IR pass –> Add threshold layer to remove parts of film image and leave just dust and scratches –> import result to opened positive file made with NLP –> turn mask into the alpha channel –> expand mask for several pixels and blur it by the same amount of pixels –> move it a little to place dust in the center of masked zones –> make content-aware fill (which is the longest step actually). And the result was literally perfect. No dust, no ugly areas which SilverFast does by removing dust. Everything is smooth and nice.

Thank you once again for the hint about how to extract the IR pass. If one day you include this option to NLP convert process – it will be even better.

P.S.: @mcbrineellis do you do photoshop dust removing the same way?

2 Likes
13

Awesome :raised_hands::raised_hands:

Let me think a little bit more on how to implement this, but I think this is a feature I can add for v3.0 later this year.

Do you know how to use actions in Photoshop? You could record this process and turn it into an action.

-Nate

14

Im pretty interested on this thread. But i’m not able to split my DNG 64RGBI from Vuescan. I’m using windows 10. Please could you tell me how?

I’m scanning as a RAW Tiff but i want the powerful of DNG with LR.

ThankYou

15

No idea, man. Thanks to Nate – he told us about tiffsplit but it was for MacOS thus how to do this with Windows – I don’t know. You can send me a DNG example, I’ll try to split it on my mac at least to see does it work with Vuescan containers.

16

Hello, i have several tests and my conclusion:

Vuescan: 64RGBI doesn’t keep IR data as TIFF
Silverfast: 64RGBI keep and can be splitted with tiffsplit

Need a good workflow to scan with VS (cheaper as SilverFast) and i wouldn’t like to use TIF, my workflow ideal would be using DNG with NLP in LR.

Could anyone give me an advice.

Thank you

17

Hello everyone,

there is an open-source program called dcraw which is used under the hood by most of the solutions listed here for Silverfast DNGs.

Silverfast
The Infrared record can be exported using the command
dcraw -W -6 -s 1 PATH_TO_YOUR_16_BIT_SILVERFAST_DNG_FILE.dng the result is a 1-channel. 16-bit .PGM file containing only the IR-record. It will be placed in the same folder as the input-file. I am not aware of a manual way of doing this using Silverfast, as I don’t own the software. I tested this solution with the Files shared by @galxcom on 64 bit Linux Mint, but it should be the same on windows/Mac.
dcraw -W -6 -s 0 PATH_TO_YOUR_16_BIT_SILVERFAST_DNG_FILE.dngwill output a 16-bit 3-channel .PPM file containing the color channels. .PPM and .PGM were created to be pretty much the most simple conceivable format for image data and along with dcraw(built using only the ANSI C standard libs) are definitely not going anywhere anytime soon.

Vuescan
If you are saving raw rgbi dng scans from Viuescan the above solution will not work. However Vuescan allows you to use the .DNG file you saved as an input and thus re-encode the file using either a RGBI tif or even just outputting a 16-bit TIF/PNG for the IR-channel. This is somewhat more cumbersome than calling a command line program(can be done for batches of files) but has the advantage of not requiring extra software.

CAVEAT:
Having said all of this, there is a fundemental flaw to using just the thresholded IR-record for dust removal, as it is correlated with the RGB records. This is because along with capturing physical imperfections in the emulsion the IR-record is also influenced(albeit faintly) by the the other channels.

Digital ICE, as outlined in the original Patent accounts for this by shifting all 4 channels to a cubic base and subtracting the R, G, B records multiplied by three different factors. These factors depend on the film, the scanner, exposure and development. Beneath an example of this difference. The first one is just the IR channel with it’s blackpoint adjusted for visibility and the second is the result of subtracting a linear combination of the RGB channels from the IR channel and subsequently adjusting the black point.

demo2
demo21920×1288 114 KB

demo
demo1920×1288 172 KB

Better results can be achieved like this however it is far from trivial to implement an automatic extraction of a map of imperfections like this. Seeing as I am trying to obtain “clean” maps of these imperfections for hundred of slides I spent a substantial amount of time implementing this in software. Currently it’s only implemented for 16 bit RGBI tifs.

Please let me know if this is something that could be useful to you, as I could write a little program with a user interface and make it available to others. Possibly supporting DNG (silverast + Vuescan) and tif.

EDIT:
While looking at libraw a C++ library to open and develop raw files I just stumbled upon one of the sample programs, called 4channels which splits both Vuescan and Silverfast DNGs into their 4 components. Some quick experimentation revealed that it doesn’t work for RGBI tifs though. This is probably easier to install/use than dcraw. It can be downloaded at Download | LibRaw

1 Like
18

Hey @nate ! Are there any updates on this?

19

You could make us happy!

20

It’s still on my agenda, but need to finish up some other pieces of development before I can tackle it. And as @mightimatti has pointed out, while it may be easy to separate out the IR channel, it is not an easy task to use it well to remove dust.

2 Likes