Can't import media from directory with certain Unicode chars

18.0.3 (might have been an issue prior but I can't really say with certainty.) MacOS Monterey 12.15.1

I have a file called filename.mp4 (or flac or whatever) sitting in a folder with a name that includes unicode char U+1D517 ( 𝔗) (in reality I've tried a bunch of similar MacOS supported Unicode characters and they all do the same thing. (also I can't just paste the Unicode character in here because it breaks the forum as well!)

I drag the file from the finder into the Media Pool. It acts like I didn't. I try to import via File/Import/Media and the same thing happens.

I change the name of the folder to a 'normal' character - "T" for example and it imports fine both by drag and drop and File/Import/Media.

Seems fully reproducible. Just a bug report. Happy to demo it live any time.

LT

I tend to use ASCII characters only in file or folder names. Old habit from UNIX times, but still very reliable.

Still a bug in 18.0.4.

FWIW this is very easy to reproduce. Again - MacOS supports Unicode as a standard part of the file naming conventions so Resolve should respect the same conventions as the supported host OS. In this case it appears to just drop or not process files with Unicode without even a message rejecting it.

L.

18.0.3 (might have been an issue prior but I can't really say with certainty.) MacOS Monterey 12.15.1

I have a file called filename.mp4 (or flac or whatever) sitting in a folder with a name that includes unicode char U+1D517 ( 𝔗) (in reality I've tried a bunch of similar MacOS supported Unicode characters and they all do the same thing. (also I can't just paste the Unicode character in here because it breaks the forum as well!)

I drag the file from the finder into the Media Pool. It acts like I didn't. I try to import via File/Import/Media and the same thing happens.

I change the name of the folder to a 'normal' character - "T" for example and it imports fine both by drag and drop and File/Import/Media.

Seems fully reproducible. Just a bug report. Happy to demo it live any time.

LT

Testing DaVinci Resolve v18.1.2.0006 (macOS/Clang x86_64)

Same issue. Attempt to import media from a valid directory that has Unicode characters simply fails to import (no error or crash). Importing the same file from the same directory with standard ASCII directory name works fine.

Debug from opening of empty project (Project 8) follows - the next thing I did was try to import an audio file from the directory with the Unicode name:

[0x7ff846da18c0] | SyManager.ProjectManager | INFO | 2022-12-30 06:24:20,657 | Loading project (Untitled Project 8) from project library (Local Database) took 314 ms
[0x7ff846da18c0] | UI.ActionManager | WARN | 2022-12-30 06:24:20,677 | Action [fairlightImmersiveGenerateImmersiveMaster] is not a valid global action
[0x7ff846da18c0] | UI.ActionManager | WARN | 2022-12-30 06:24:20,677 | Action [fairlightImmersiveRendererSettings] is not a valid global action
[0x7ff846da18c0] | UI | WARN | 2022-12-30 06:24:20,736 | Failed to find value '0' in combo-box
[0x7ff846da18c0] | SyManager.Gallery | INFO | 2022-12-30 06:24:20,744 | Start purging still caches
[0x7ff846da18c0] | SyManager.Gallery | INFO | 2022-12-30 06:24:20,745 | Finish purging still caches
[0x7ff846da18c0] | SyManager.Signals | INFO | 2022-12-30 06:24:20,763 | Media pool relink status changed to 0
[0x7ff846da18c0] | SyManager.Signals | INFO | 2022-12-30 06:24:20,796 | Current project pointer (Untitled Project 8) is changed
[0x7ff846da18c0] | SyManager.Signals | INFO | 2022-12-30 06:24:20,796 | Current timeline pointer () is changed
[0x7ff846da18c0] | Undefined | INFO | 2022-12-30 06:24:20,802 | The buttonId [7] is not found for [0] in [0]
[0x7ff846da18c0] | UI | INFO | 2022-12-30 06:24:20,812 | UploadWipeData called before UiGPU has been initialized. Returning false
[0x7ff846da18c0] | UI | INFO | 2022-12-30 06:24:20,813 | Gallery pointer is changed, refreshing color gallery
[0x7ff846da18c0] | UI | INFO | 2022-12-30 06:24:20,830 | Gallery pointer is changed, refreshing gallery browser
2022-12-30 06:24:20.847 Resolve[37949:2144896] Invalid state set [-100].
[0x7ff846da18c0] | SyManager.Signals | INFO | 2022-12-30 06:24:20,853 | Main view page is changed to 1
[0x7ff846da18c0] | UI | WARN | 2022-12-30 06:24:20,853 | Failed to get auto update information.
[0x7ff846da18c0] | SyManager.Signals | INFO | 2022-12-30 06:24:20,885 | Main view page is changed to 17
[0x7ff846da18c0] | UI | WARN | 2022-12-30 06:24:20,982 | UI Persistence - MediaPoolFloatingWindowGeometry not read
[0x7ff846da18c0] | SyManager.Signals | INFO | 2022-12-30 06:24:21,009 | Main view page is changed to 17
[0x7ff846da18c0] | UI | INFO | 2022-12-30 06:24:21,012 | Launching main window
### DEL_DBG IOFileSys::DeleteDirFully - command = rm -rf '/Users/levine/Movies/Resolve Project Backups/f9381674-2bf0-410e-a6eb-d21ddb38ea85'
### DEL_DBG IOFileSys::DeletePath[1471] p_OSErrCode = 0, p_ErrMsg =
[0x7ff846da18c0] | SyManager | WARN | 2022-12-30 06:24:21,495 | No reply received from file system, assume successfully deleted folder /Users/levine/Movies/Resolve Project Backups/f9381674-2bf0-410e-a6eb-d21ddb38ea85.
### DEL_DBG IOFileSys::DeleteDirFully - command = rm -rf '/Users/levine/Movies/CacheClip/f9381674-2bf0-410e-a6eb-d21ddb38ea85'
### DEL_DBG IOFileSys::DeletePath[1471] p_OSErrCode = 0, p_ErrMsg =
[0x7ff846da18c0] | SyManager | WARN | 2022-12-30 06:24:21,506 | No reply received from file system, assume successfully deleted folder /Users/levine/Movies/CacheClip/f9381674-2bf0-410e-a6eb-d21ddb38ea85.
### DEL_DBG IOFileSys::DeleteDirFully - command = rm -rf '/Users/levine/Movies/.gallery/f9381674-2bf0-410e-a6eb-d21ddb38ea85'
### DEL_DBG IOFileSys::DeletePath[1471] p_OSErrCode = 0, p_ErrMsg =
[0x7ff846da18c0] | SyManager | WARN | 2022-12-30 06:24:21,516 | No reply received from file system, assume successfully deleted folder /Users/levine/Movies/.gallery/f9381674-2bf0-410e-a6eb-d21ddb38ea85.
2022-12-30 06:24:36.585 Resolve[37949:2144896] +[CATransaction synchronize] called within transaction
2022-12-30 06:24:36.814 Resolve[37949:2144896] +[CATransaction synchronize] called within transaction
2022-12-30 06:24:42.327 Resolve[37949:2144896] Invalid state set [-100].

Don’t post that googlemoogle, but a proper log file for the developers to look at.
If you don’t know how, check the FAQs.

Similarly - I am finding I can create an unrecoverable loop (requiring a force quite, best I can tell) by doing the following:

Create a new project called Unicode U+1F919 (the call me hand emoji). (Resolve will happily accept it or other Unicode characters)
Try to Save it.

I am on: DaVinci Resolve v18.1.2.0006 (macOS/Clang x86_64) MacOS 13.1

Generally I'm finding that Unicode is partially supported - but it seems like while it will display in Resolve - your file system handling of it is quite broken - even on a file system which fully supports it (like MacOS 13.1 (22C65)). It seems that timelines, projects, file names, etc. are all fine - but when resolve interacts with the filesystem things go sideways. Nothing in particular of interest in the logs, but the results are obvious to even a cursory attempt to check for the problem. Screenshot showing some uses that work fine attached. If you change the actual project name to include the unicode character and then save it, all hope is lost.

For what it's worth - I'm finding these issues in real world scenarios where I'm receiving filenames, zip files, etc. with Unicode in them.

Thanks.

LT

P.S. Including the actual Unicode U+1F919 in this post also causes a SQL Error in the Bulletin Board system.

Tested with 18.1.4. Same results. Bug still unfixed.

Just tested on 18.5B Build 31 and the same bugs persist.

I had the same issue on 18.5. Brand new computer and brand new DR version and the first thing it does is not import files. I tried installing the version I had been using on my old computer (16.2.8) and it had the same issue. On a whim, I deleted the emoji's in the file name. That did it. Jeez Loiusse. I mean, it could at least complain about the file name or something.

Anyhoo, the upshot is, for anyone reading this thread in the future, instead of the file being bad or unsupported, it might just be unsupported characters the file name.

Yup. It's not that there's no workaround with enough effort, it's just that since modern OSs support UTF-8 to not do so k and so ungracefully) is a bug, so I keep testing, hoping the bug gets resolved. At least I'm not all alone in experiencing it.

Just tested 18.6.3 Build 19 and have the same results.

Same bug is still present in Resolve Studio 18.6.5 on Windows 11.

I am using specific unicode characters in file names to indicate script based processing status on my media files.

For example a script adds:
- "Pushpin" symbol to filename if it has location string added based on GPS coordinates.
- Or "triangle" symbol is added if video file has been re-compressed (isn't the original source).

Importing filenames that have such unicode characters fails on Davinci Resolve Studio. No visible error, the file just never appears. 100% replicatable and same files work if the unicode characters are removed.

How to replicate:
1) Name one media with such unicode characters in filename
2) Try to import that file in Davinci Resolve Studio
3) Nothing happens and the file doesn't show up

In davinci_resolve.log file error like this is logged:

[0x0000211c] | IO | ERROR | 2024-02-08 17:27:31,968 | GetFileSizeEx error 6
[0x0000211c] | IO | ERROR | 2024-02-08 17:27:31,968 | Failed to create decoder for I:\2024.01.20 140635 Uunisaaret, Helsinki ?????????.mp4
[0x0000211c] | IO | ERROR | 2024-02-08 17:27:31,968 | Failed to open video file <I:\2024.01.20 140635 Uunisaaret, Helsinki ?????????.mp4> for decoding, error: <Codec is not opened yet.>
[0x0000211c] | IO | ERROR | 2024-02-08 17:27:31,968 | Failed to read video track information from file <2024.01.20 140635 Uunisaaret, Helsinki ?????????.mp4> in I:\, track 0: codec DUMMY is not supported

Just tested on Mac OS with Resolve 19.0B Build 20 and have the same issue. Quite disappointing as you can't even load files from a directory with the unicode in it, despite being fully supported by the OS and seen by Resolve without issue. It just silently fails. I'll keep checking but not holding my breath.

19B build 25 - same results.

If there's any hope for a fix, it will get more of a chance if you post this to the beta subforum.

There seems to be an “official exception” for unicode characters in the range (edited from U+1D500 to) U+1D400-U+1D7FF.

The following article:

https://en.wikipedia.org/wiki/Mathemati ... ic_Symbols

Or:

https://de.wikipedia.org/wiki/Unicodebl ... he_Symbole

Contains these key phrases:

“Unicode now includes many such symbols (in the range U+1D400–U+1D7FF“

And:

“Unicode expressly recommends that these characters not be used in general text”

This does sound, as if it explains the woes, you’re experiencing. Correct me, if I’m wrong.

Regards

I got excited when I read your response because I was hoping it was a rational explanation for the behaviour, and that perhaps I had just gotten unlucky with the Unicode characters that I was originally given to work with, however, it's not quite that simple.

I just tried a bunch of different characters that are all supported by MacOS in file/directory naming and many more than just that range neither work, nor provide the user any indication of them not working, or why.

U+1D122, U+11700, U+10153 are just a few examples (and picked quite randomly)

I can't identify a pattern as to which will, or won't, work, but it's not just a simple range, and expecting users to figure out which work (and more importantly don't work) is not reasonable. I'd argue that if the supported OS is happy to use a character, that the application running on it should as well, particularly for something like interaction with the filesystem.

A couple of real world examples:

1) Make a directory that contains one of the Unicode characters that don't work, put a simple media file (a wav file for example) with a simple name (test.wav for example) in it. Then try to drag that test.wave file from the finder into the Resolve Media Pool for a project. It just doesn't do anything, same if you drag the whole directory in fact it just seems to ignore all of the assets in the directory with the name containing a character it doesn't like. This means if you dragged a directory with a bunch of assets in it, in folders, some of which contained these characters, and some which did not, you wouldn't be able to predict which files would show up in the pool and which would just be silently ignored.

Directory structure with one directory that has a 'bad' character, and the result from dragging the top level directory into the Media Pool.

Directory and result in pool.png (203.43 KiB) Viewed 417 times

2) Go into the Media Storage panel in the workspace and look at the directorys listed - interestingly it seems to not properly render the ones using characters it 'doesn't like', instead showing a Black Diamond with a white question mark in it (0xFFFD I think) and nothing seemingly in that folder, even if there is something there. (As an amusing aside, it's quite happy for that character - the replacement character - to be used in the filenaming.)

Anyway - I'm only stuck on this issue because I like predictable user experiences, and I ran head long into this in the real world and it was frustrating and confusing, so I've been taking the time to put it forward in the hopes that one of the team sees it and addresses it to save the next user from the confusion.

I'm going to just humbly keep pushing for recognition of, at least, the set of characters recognized natively by the OS being supported.

It does not alter the fact, that the range of characters cannot be counted on being available in general use. Period. As clearly recommended by the standards authority. Period.

File system are especially complicated cases, where a file name in one system, should be usable in a different system (within some given limitations, that also vary from OS to OS sometimes version to version). Not only in one language. Or honoring one Companys or individuals whims.

How big are the chances, that your insistence will find willing ears, investing time and effort in making your preferences in all new OS’es, maybe patching existing systems, and start a concerted and extremely costly effort to modify - and test - all standard libraries in programming languages to ignore even some of the both valid and logic reasons behind the current restrictions?

Unicode has been “Gods gift” to programmers using standard, well tested and very reliable text handling routines and libraries (including case conversion, maybe even some types of language ‘transliteration’ in a trustworthy and reproducible way; including searching, sorting and all that jazz. Etc!!!

On top of that, you need “fonts” - sets of letters, that can actually be reproduced on printers, screens etc. In a reliable, approved and standard way. The “standard” fonts are not required to cover the areas, you want included. There are special fonts for special purposes and requirements - e.g. mathematics - that serve these use cases, with the proviso, that the use of these characters explicitly is NOT for use in general text.

Imagine using EU’s 24 official and legally binding languages (with different character sets to boot) in the old Codepage system, or the far worse approach used by many Americans, where “ø” and “ö” becomes “o”. Search routines would deliver results worse, than Googles “paid-for” primary results on the first pages.

Or even worse, letting company or individual special to unique fashion interests willy-nilly define actual valid alfabetic content like in the total “emoji” chaos?

You may insist to use characters, that are explicitly recommended by the standard authorities NOT to be used in general text (and filenames certainly are covered by that). Nobody is preventing you from wasting your time on a loosing battle. But you also risk wasting other people’s time, maybe causing misunderstandings, bad search results etc. in the wake of your quest to deliberately ignore the consequences of not honoring existing standard advice.

Everybody needs a hobby and a pet peeve. Alas…

Regards

I hear what you are saying, but the 'range of characters' having the problem is not limited to the characters that are in the U+1D400-U+1D7FF set. If it were I would just chalk it up to being a decision not to recognize them for the reasons you stated.

The reality is that there are many characters not being recognized and they seem to be all over the unicode map.

As to the OSs and filesystems being a guide. We are in the here and now. There are three supported OSs. Mac OS, Windows, and Linux. Of these Linux is the only one where normal users are making file system choices generally. Windows and Mac users just use the underlying filesystems Microsoft and Apple provide. It doesn't seem like a very big ask to have something interacting with those specific file systems to be compatible with the character sets they support at the time of release. I don't have a Windows or Linux machine setup with Resolve but I'd be very curious if the same problem exhibited on them as well.

So - I'm not really on a "quest to deliberately ignore the consequences of not honoring existing standard advice." I'm just wanting things to work nicely together without having to refer back to standards - and in this case the standards may say that the range you pointed out shouldn't be used but as I indicated (and only as arbitrary examples) U+1D122, U+11700, U+10153, etc. don't work and are not in that range.

Also - to better define the playing field. Premier Pro handles exactly the same naming without exception or issue. Although to Resolve's credit, Premier doesn't know what to do with flac files (don't get me started on that one) so I did the test with three wav files instead.

An example of the same characters being handled fine in Premier Pro

Premier Pro 2024 with the same File System names.png (595.84 KiB) Viewed 361 times

FWIW I ran into this because a real world user provided me a bundle of files that were 'creative' in their names. Creativity should be expected in creative the arts fields. The 'chaos' of it doesn't bother me at all, because it's actually limited to only the chaos supported by the underlying tools (in this case the OS the user used to put the bundle together in.) I dislike when technology (and rigid engineering) specifically tries to stifle creativity, and even more so when there isn't a good reason to do so. So - I guess at the end of the day my view on the chaos is: "Bring the noise!"

L.

It does not alter the fact, that the range of characters cannot be counted on being available in general use. Period. As clearly recommended by the standards authority. Period.

Not sure where you draw that conclusion from. Just because the Unicode 13 standard says "The characters in this block are intended for use only in mathematical or technical notation, and not in nontechnical text." doesn't mean that they can't be "counted on" being available. You definitely are able to use them anywhere where you think using mathematical notations make sesnse, you may even use them in filenames related to mathematical notations if that makes sense to your use case. (Although I can't imagine such use cases myself)

File system are especially complicated cases, where a file name in one system, should be usable in a different system (within some given limitations, that also vary from OS to OS sometimes version to version). Not only in one language. Or honoring one Companys or individuals whims.

...

Indeed, that is literally why UTF was invented 30 years ago, and why every operating system supports it: You don't have to know what characters each operating system supports or not, you don't have to worry about supporting regional character sets, text sorting or anyting like that because its all happening via the standardized libraries that have been widely supported by every OS and programming library for almost 20 years now.

That is why DaVinci Resolve is the outlier here that is causing problems and should be fixed.

EVERY operating system out there supports unicode characters in filenames. Windows, MacOS, Linux, Android, iOS, your TV operating systems, your game consoles. Throw unicode filenames at them and they all work just fine.

Same with basically every other piece of software that isn't in "legacy" state without updates for decades. You can use unicode filenames in every photo editor I know, every other video editor I know, every single piece of software that deals with saving or loading files works just fine with unicode characters. It is only DaVinci resolve and their over decade old file handling api that is having problems and absolutely should be updated to avoid such issues.

You have your fun hunting non-visible, non-supported, non-standard and/or non-existing characters.

The text may be OK, but if the installed font-set does not include a way to represent/print the specific character, something else is done, shown, printed or not (down to a space or even missing space for the character - defined as having the width zero in the font-set - on a print).

There are even people, who deliberately install “inferior” font-sets (that they deem to be gorgeous), that do not cover even the most used standard characters in their region, and then complain that a text - maybe requiring the use of the common Greek alphabet - cannot be represented on screen. The text still exists in its original unicode text string, and will work as intended in standard environments within the defined standard limits. Period.

Whatever is the case in your environment, and you even deliberately ignoring standard advice on characters best avoided in general use text, does not invite support.

I’m not continuing this discussion. It’s a waste of my time in my view.

I'm happy to not continue the discussion with you, because you are clearly not interested in the problem or solutions to it. Period. You seem to be just interested in pretending to point to standards, but as long as you are doing that; where do you see a standard that shows Unicode U+10153 (that I used in my recent examples) as one of the "characters best avoided in general use"?

At any rate - Windows, Mac, and Linux all appear to support many characters that are under the Unicode standard (https://www.unicode.org/versions/Unicode15.1.0/) in their generally supported file systems. At least one significant competitive product (Premier Pro) to Resolve also seems to support them (see my actual example). Whatever your view is on the chaos of Unicode doesn't really matter because real users can, will, and do use a variety of Unicode characters, so good software will support it and good developers will see the issue as a legitimate one, though perhaps not important enough to give a priority in the backlog of actual work that needs to be done by finite resources.

Please don't bother to reply (unless you can actually reference part of the Unicode standard saying not to use Unicode characters like U+10153.) Period.

Have a nice day!

“Unicode expressly recommends that these characters not be used in general text”

In our client turnover specs for edit materials and files, we specifically tell them:

1) no spaces in file names

2) no punctuation except - or _

3) specially avoid / \ | []() # % <> — – (no emdashes, no endashes)

4) specially avoid all high-ASCII characters (umlauts, accents, and so on)

We warn them, "ignore these naming-procedures at your own risk. We are not responsible for delays or increased hours with problematic workflow formats, file names, and drive issues."

Real cameras (Alexa, Blackmagic, most Canon cinema, Red, Sony Venice) generally do something like A001_C001_05026M.codec, with the codec being R3D or XEVC or Arriraw or ProRes or whatever. Those file names are never a problem.

Amen, bro.
Part of what we hand to students for the video labs.

I got excited when I read your response because I was hoping it was a rational explanation for the behaviour, and that perhaps I had just gotten unlucky with the Unicode characters that I was originally given to work with, however, it's not quite that simple.

I just tried a bunch of different characters that are all supported by MacOS in file/directory naming and many more than just that range neither work, nor provide the user any indication of them not working, or why.

U+1D122, U+11700, U+10153 are just a few examples (and picked quite randomly)

I can't identify a pattern as to which will, or won't, work, but it's not just a simple range, and expecting users to figure out which work (and more importantly don't work) is not reasonable. I'd argue that if the supported OS is happy to use a character, that the application running on it should as well, particularly for something like interaction with the filesystem. Cut...

Last message from me in this thread. General comment illustrating a point.

ONLY regarding "ordinary text", not touching on filename use in any way (filenames should be regarded as "symbols" representing A content - not a full mathematical, numismatic or historical thesis. Most chinese - as an example - forced to use the latin alfabet in some circumstances, filenames mainly, certainly see the naming as pure "symbols", since most don't even "use" the latin alfabet - restricted values or not).

Why should U+1D122 (musical symbol F clef, see: https://www.compart.com/en/unicode/U+1D122), U+11700 (Atom Letter Ka, see: https://www.compart.com/en/unicode/U+11700), U+10153 (Greek Acrophonic Attic Five Hundred Staters, see: https://www.compart.com/en/unicode/U+10153) just picked quite randomly (eehhh…???) be universally supported or their "MIA" state even need explanation?

We’re now entering the realm of the unicode “Supplementary Multilingual Plane (U+10000 - U+1FFFF)” also containing the aforementioned mathematical symbols (certainly to be avoided in ordinary text), Domino Tiles, Playing Cards, Emoticons, Ornamental Dingbats, Alchemical Symbols and whatnot). See: https://en.wikipedia.org/wiki/Plane_(Unicode)#Supplementary_Multilingual_Plane.

Maybe, just maybe, the actual name of the "Unicode Plane" (and unicode number range) should and could be used as a hint toward logic, not expecting full, general and universal coverage of these 65536 characters and their "display images" in fonts (software, firmware or ROM). It would of course require a willingness to look beyond personal peeves based on just the name "unicode" and enter the realm of real life requirements and documentation (where real life gets messy).

I certainly can understand, that not even arabic script used in countries just across the mediteranian will see the universal need for inclusion of - let’s say - Aegean Numbers (see: https://en.wikipedia.org/wiki/Aegean_Nu ... code_block)) within the Supplemental Multilingual Plane. Thai and Chinese writers probably would have even more specialized use cases.

Even a slightly hidden demand for universal unicode support of every defined character everywhere is in my view a trifle…. ehh… bonkers (for lack of "printable" words in this context).

Regards

P.S. If anyone is finding Apples (or Microsofts, Adobes, Linux etc) support of unicode characters in various environments hard to swallow, why not do the proper thing: Contact the companies, and try to convince Apple, Microsoft, Adobe etc. and the unicode standard handlers to make all defined unicode characters mandatory. Good luck.