Add GoString and CString functions to match C package API

[TotallyGamerJet]

When using the the special C package there are a few functions that convert from Go types to C types and vice versa. When porting code that uses C it's annoying not having an equivalent to these functions. Especially, GoString and CString which convert a C string to a Go string and vice versa. Even ebitengine had to have it's own implementation and writing these over and over is begging for mistakes. Purego already uses it's own internal versions.

I purpose adding these two functions to purego with the following signatures.

// Cstring converts a Go string to a null-terminated C string.
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling free.
func CString(string) *byte

// GoString converts a null-terminated C string and copies it into
// a Go allocated string.
func GoString(*byte) string

I believe these are the most used functions so should be added first. If there is strong demand later for the other ones a new issue can be created.

[hajimehoshi]

Does CString add a null-termination automatically?

// It is the caller's responsibility to arrange for it to be
// freed, such as by calling free.

How does a user call free with purego?

[TotallyGamerJet]

Does CString add a null-termination automatically?

Yes. That's what the C package does. Matching it's behavior would be the least surprising

How does a user call free with purego?

var free func(unsafe.Pointer)
purego.RegisterFunc(&free, freeHandle)
free(unsafe.Pointer(cStr))

And freeHandle can be gotten using Dlsym

[hajimehoshi]

Yes. That's what the C package does. Matching it's behavior would be the least surprising

Sure, I'm fine if this is explicitly documented.

EDIT: I missed that the comment already said so: "Cstring converts a Go string to a null-terminated C string." Thanks!

And freeHandle can be gotten using Dlsym

So do we dlopen libc?

[TotallyGamerJet]

So do we dlopen libc?

I don't think purego should export free nor libc in its API. How someone wants to free it is up to the user

[hajimehoshi]

I don't think purego should export free nor libc in its API. How someone wants to free it is up to the user

Doesn't this depend on how CString allocates the string data?

[hajimehoshi]

If CString uses malloc from libc, a user would have to use free from libc. I'd like to know your plan how to implement CString.

[TotallyGamerJet]

If CString uses malloc from libc, a user would have to use free from libc.

That is true. I'm not sure what signature to use though so leaving it up to the user seems best. Does it take a *byte since that's what CString returns or unsafe.Pointer but then there are a bunch of unsafe.Pointer casts everywhere? I think part of purego is to make it more type safe to call into C not less.

Here is an example of what I was thinking.

func getSystemLibrary() string {
	switch runtime.GOOS {
	case "darwin":
		return "/usr/lib/libSystem.B.dylib"
	case "linux":
		return "libc.so.6"
	default:
		panic(fmt.Errorf("GOOS=%s is not supported", runtime.GOOS))
	}
}

var malloc func(size int) *byte

func init() {
	libc, err := purego.Dlopen(getSystemLibrary(), purego.RTLD_LAZY|purego.RTLD_GLOBAL)
	if err != nil {
		panic(fmt.Errorf("could not open libc: %w", err))
	}
	purego.RegisterLibFunc(&malloc, libc, "malloc")
}

// CString converts a Go string to a null-terminated C string.
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling free.
func CString(s string) *byte {
	if len(s) == 0 {
		return nil
	}
	size := len(s) +1
	c := unsafe.Slice(malloc(size), size)
	copy(c, s)
	return &c[0]
}

[hajimehoshi]

There is no way for users to know how to free the allocated memory, other than knowing what DLL/so is used, right? I don't think this is a good API...

By the way, does CString have to allocate the memory on C heap, not Go heap? If so, I was wondering why.

[TotallyGamerJet]

Perhaps then returning a Free function that frees the string?

func CString(s string) (cStr *byte, free func())

By the way, does CString have to allocate the memory on C heap, not Go heap? If so, I was wondering why.

It's the same reason for Cgo. If the C code retains the pointer but the Go code thinks it needs to be garbage collected (since it can't see into the C code) then memory corruption can happen. It's safe to pass a Go string into C if the C code doesn't keep the pointer.

[hajimehoshi]

Perhaps then returning a Free function that frees the string?

I think this makes sense, though this would vary from the original Cgo's CString.

It's the same reason for Cgo. If the C code retains the pointer but the Go code thinks it needs to be garbage collected (since it can't see into the C code) then memory corruption can happen. It's safe to pass a Go string into C if the C code doesn't keep the pointer.

As the users have to control freeing the value, would this work for example?

func CString(s string) (*byte, func()) {
    bs := []byte(s)
    if len(bs) == 0 || bs[len(bs)-1] == 0 {
        bs = append(bs, 0)
    }
    return &bs[0], func() { runtime.KeepAlive(bs) }
}

[TotallyGamerJet]

I think this makes sense, though this would vary from the original Cgo's CString.

True. I'm not sure there is a nice way to free while keeping the original API.

As the users have to control freeing the value, would this work for example?

I don't see why not since either way it is the responsibility of the caller to ensure the string is freed properly. Although what happens if the free function is ignored (cStr, _ = CString(...))? I do think it's better to make([]byte, len(s)+1) instead of casting since casting then appending is likely to be two allocations instead of one.

Perhaps, something like this?

func CString(str string) (*byte, func()) {
        var b []byte
        if len(str) == 0 {
               b = []byte{0x00}
	} else if hasSuffix(str, "\x00") {
                b = []byte(str)
	} else {
                b = make([]byte, len(str)+1)
                copy(b, str)
        }
	return &b[0], func(){ runtime.KeepAlive(b) }
}

Also, do we want the implementation to turn an empty string into a C string that only has null character instead of returning nil?

[TotallyGamerJet]

Although. What if the C function never releases its access to the string? Shouldn't it than be allocated on the C heap?

[hajimehoshi]

I don't see why not since either way it is the responsibility of the caller to ensure the string is freed properly. Although what happens if the free function is ignored (cStr, _ = CString(...))?

I see. Then what about keeping the string in a global slice or a global map in Go?

I do think it's better to make([]byte, len(s)+1) instead of casting since casting then appending is likely to be two allocations instead of one.

My code snippet is just a prototype so if there is a more efficient way, let's adopt this :-)

Also, do we want the implementation to turn an empty string into a C string that only has null character instead of returning nil?

Not sure. Let's sync with the original CString.

Although. What if the C function never releases its access to the string? Shouldn't it than be allocated on the C heap?

I don't see the point. Freeing is still the user's responsibility, right?

[hajimehoshi]

For example:

func CString(str string) (*byte, func()) {
        var b []byte
        if len(str) == 0 {
               b = []byte{0x00}
	} else if hasSuffix(str, "\x00") {
                b = []byte(str)
	} else {
                b = make([]byte, len(str)+1)
                copy(b, str)
        }
        key := uintptr(unsafe.Pointer(&b[0]))
        // TODO: Need to protect this by a mutex
        globalStringMap[key] = b
	return &b[0], func(){ delete(globalStringMap, key) }
}

[TotallyGamerJet]

I just remembered why the string needs to be on the C heap. The reason is because of Go's growing and shrinking stack. If C saves the string and then later Go grows the stack the C string pointer will point to random memory. This is bad.