Does it support utf8?

[Ailtop]

Load utf8 string, display incorrectly

[tilkinsc]

Could you provide more information? A reproducible bit of code at least would be helpful to understand what you are trying to do.

[Ailtop]

var L = luaL_newstate();
luaL_openlibs(L);
{
// luaL_dostring(L, "config = {'您好'}"); // Direct dostring is correct
luaL_dofile(L, "D:\main.lua"); // but dofile is not correct, file encoding is UTF-8, file content is "config = {'您好'}"
lua_getglobal(L, "config");
lua_pushinteger(L, 1);
lua_gettable(L, -2);
Console.WriteLine(lua_tostring(L, -1));
}
lua_close(L);

[tilkinsc]

This is weird behavior because that directly calls a native function within the dll library. What version of Lua were you using?

[Ailtop]

Lua53

[Ailtop]

luaL_dostring(L, File.ReadAllText("D:\main.lua")); // this is also correct
It looks like the dofile is incorrect.

[tilkinsc]

Seems like in lua53 luaL_dofile will call luaL_loadfile which calls luaL_loadfilex (just like lua) with mode null which calls lua_load within the dll library. Yada yada if it fails it leaves an error on the stack. Could you tostring the error, perhaps that will give us more information. https://www.lua.org/manual/5.3/manual.html#lua_load

Also more information about mode,
The string mode controls whether the chunk can be text or binary (that is, a precompiled chunk). It may be the string "b" (only binary chunks), "t" (only text chunks), or "bt" (both binary and text). The default is "bt".

[Ailtop]

There is no error in luaL_dofile.
I've given you all the code to try out.

[tilkinsc]

I got around to trying it out, and have confirmed the issue. Investigating.

[tilkinsc]

..................

Can't fix without modifying source...... (which means rolling custom code lib-side D:)

This confirms the issue with lua_loadstring

var L = luaL_newstate();
luaL_openlibs(L);
{
	byte[] bytes = File.ReadAllBytes("./main.lua");
	string str = Encoding.UTF8.GetString(bytes);
	luaL_loadbufferx(L, str, (ulong)str.Length, "chunk", "bt");
	for (int i=0; i<bytes.Length; i++)
	{
		Console.Write($"{bytes[i]:x} ");
	}
	lua_call(L, 0, 0);
	lua_getglobal(L, "config");
	lua_pushinteger(L, 1);
	lua_gettable(L, -2);
	string? str2 = lua_tostring(L, -1);
	Console.WriteLine(str2);
}
lua_close(L);

As for luaL_dofile, there's something emitting incorrect data lua-side. Lua is capable of UTF-8 and UTF-8 BOM, but apparently this code path isn't. The safest way to load code is using {luaL_loadbuffer,,x} directly. I confirmed this is a Lua-owned bug by using my REPL app capable of ordered file/string execution and the fault transpires there as well in the same exact way.

I am open to anyone's contributions for the solution!

Temporary Patch:

// does not handle stdin
// does not handle shebangs maybe i.e. #!/bin/lua
// not sure if it handles compiled lua?
public static int _luaL_dofile(lua_State L, string fn)
{
	string script = File.ReadAllText(fn);
	int status = luaL_loadbufferx(L, script, (ulong) script.Length, $"@{fn}", "bt");
	if (status > 0)
		return status;
	return lua_pcall(L, 0, LUA_MULTRET, 0);
}