Skip to content
This repository was archived by the owner on Dec 15, 2025. It is now read-only.
This repository was archived by the owner on Dec 15, 2025. It is now read-only.

Load nweight-user-features as resource file #745

@jnsrnhld

Description

@jnsrnhld

Hi,

I wondered why the sparkbench/graph resource nweight-user-features is not loaded as a resource. The resource is packaged together with the .jar anyway, so I'd propose something like this:

NWeightDataGenerator.scala

[...]
def loadModel(modelPath: String, sc: SparkContext, partitions: Int): MatrixFactorizationModel = {
    getClass.getResourceAsStream(modelPath) match {
      case null => throw new FileNotFoundException(modelPath)
      case stream =>
        val in = new DataInputStream(stream)
        val weights = new Array[(Int, Double)](MAX_ID)
        for (i <- weights.indices) {
          val w = in.readFloat()
          weights(i) = (i, w)
        }
        in.close()

        val userFeatures = sc.parallelize(weights, math.max(400, partitions)).map { case (i, w) =>
          (i, Array(w))
        }
        val userFeaturesPartitioned = userFeatures.partitionBy(new HashPartitioner(partitions))
        userFeaturesPartitioned.cache()
        // Model matrix is symmetric, so productFeatures is the same with userFeatures
        new MatrixFactorizationModel(1, userFeaturesPartitioned, userFeaturesPartitioned)
    }
 }
[...]

and the workload config:

nweight.conf

[...]
hibench.nweight.model_path	            /nweight-user-features
[...]

This could be beneficial in cases where the execution environment does not contain the hibench directory itself.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions