add shared guard for setkey

NEWS.md

@@ -54,6 +54,8 @@
 
 10. `setkey()`, `setindex()`, `CJ()`, and `setnames()` now prevent creating ambiguous keys from duplicated column names, and keyed joins now error on existing duplicated key columns rather than silently giving incorrect results, [#4888](https://github.com/Rdatatable/data.table/issues/4888) and [#4891](https://github.com/Rdatatable/data.table/issues/4891). Thanks @magerton and @MichaelChirico for the reports, and @ben-schwen for the fix.
 
+11. `setkey()` no longer modifies the original data.table when called on a shallow copy (e.g. `DT1 = DT[TRUE]`; `setkey(DT1, col)` was silently reordering `DT` in-place as well, while retaining the key), [#5230](https://github.com/Rdatatable/data.table/issues/5230). Thanks @daynefiler for the report and @ben-schwen the fix.
+
 ### Notes
 
 1. {data.table} now depends on R 3.5.0 (2018).

src/reorder.c

@@ -66,7 +66,14 @@ SEXP reorder(SEXP x, SEXP order)
   void *TMP = R_alloc(nmid, maxSize);
 
   for (int i=0; i<ncol; ++i) {
-    const SEXP v = isNewList(x) ? VECTOR_ELT(x,i) : x;
+    SEXP v = isNewList(x) ? VECTOR_ELT(x,i) : x;
+    // If this column is shared with another R object (e.g. a shallow-copied table),
+    // materialise a copy first so the in-place memcpy below only affects this
+    // data.table and not the original. Fixes #5230
+    if (isNewList(x) && MAYBE_SHARED(v)) {
+      v = copyAsPlain(v, -1);
+      SET_VECTOR_ELT(x, i, v);
+    }
     const size_t size = RTYPE_SIZEOF(v);    // size_t, otherwise #61 (integer overflow in memcpy)
 
     switch (size)